MPI-Flow: Learning Realistic Optical Flow with Multiplane Images

Novel View Image from MPI. To render realistic image $\textbf{I}_{t}$ under a target viewpoint $\bm{\mu}_{t}$, we use pixel warping from the source-view MPI in a differentiable manner. Specifically, we use a neural network $\mathcal{F}$ as in [13] to construct $N$ fronto-parallel RGB$\sigma$ planes under source viewpoint $\bm{\mu}_{s}$ with color channels $\textbf{c}_{n}$, density channel $\bm{\sigma}_{n}$, and depth $\textbf{d}_{n}$ from the input image $\textbf{I}_{s}$ and its depth map $\textbf{D}_{s}$ as:

$$\{(\textbf{c}_{n},\bm{\sigma}_{n},\textbf{d}_{n})\}^{N}_{n=1}=\mathcal{F}(\textbf{I}_{s},\textbf{D}_{s}),$$

(1)

where $N$ is a predefined parameter that represents the number of planes in MPI. Each pixel ($x_{t}$, $y_{t}$) on the novel view image plane can be mapped to pixel ($x_{s}$, $y_{s}$) on $n$-th source MPI plane via homography function [15]:

$$\left[x_{s},y_{s},1\right]^{T}\sim\textbf{K}\left(\textbf{R}-\frac{\textbf{t}\textbf{n}^{T}}{\textbf{d}_{n}}\right)\textbf{K}^{-1}\left[x_{t},y_{t},1\right]^{T},$$

(2)

where R and t are the rotation and translation from the source viewpoints $\bm{\mu}_{s}$ to the target viewpoints $\bm{\mu}_{t}$, K is the camera intrinsic, and $\textbf{n}=[0,0,1]$ is the normal vector. Thus, the color $\textbf{c}^{\prime}_{n}$ and density $\bm{\sigma}^{\prime}_{n}$ of each new plane for the novel view $\textbf{I}_{t}$ can be obtained via bilinear sampling. We use discrete intersection points between new planes and arbitrary rays passing through the scene and estimate integrals:

$$\textbf{I}_{t}=\sum_{n=1}^{N}\left(\textbf{c}_{n}^{\prime}\bm{\alpha}_{n}^{\prime}\prod_{m=1}^{n-1}\left(1-\bm{\alpha}_{m}^{\prime}\right)\right),$$

(3)

where $\bm{\alpha}^{\prime}_{n}=\exp\left(-\bm{\delta}_{n}\bm{\sigma}^{\prime}_{n}\right)$ and $\bm{\delta}_{n}$ is the distance map between plane $n$ and $n+1$ and we set the initial depth of MPI planes uniformly spaced in disparity as in [13].

Optical Flow from MPI. Although MPI-based methods synthesize realistic images, reliable optical flow maps are also needed to train learning-based optical flow estimation models. Therefore, we propose adding an additional optical channel in each plane. To this end, we compute the optical flow on the $n$-th plane at pixel $[x_{s},y_{s}]$ of source image $\textbf{I}_{s}$ by $\textbf{f}_{n}=[x_{t}-x_{s},y_{t}-y_{s}]$ with a backward-warp process in terms of the inverse equivalent form of Equation (2):

$$\left[x_{t},y_{t},1\right]^{T}\sim\textbf{K}\left(\textbf{R}^{\dagger}-\frac{\textbf{t}^{\dagger}\textbf{n}^{T}}{\textbf{d}_{n}}\right)\textbf{K}^{-1}\left[x_{s},y_{s},1\right]^{T},$$

(4)

where $x_{s}$ and $y_{s}$ are uniformly sampled from a $H\times W$ grid. $\textbf{R}^{\dagger}$ and $\textbf{t}^{\dagger}$ are the inverses of R and t, respectively.

Independent Object Motions. To model more realistic motions, we propose applying separate virtual motions to objects and static backgrounds extracted from the scene. Therefore, we utilize an instance segmentation network $\Omega$ [7] for extracting the main object in the source image $\textbf{I}_{s}$ as:

$$\textbf{M}_{s}=\Omega(\textbf{I}_{s})\in\mathbb{R}^{H\times W},$$

(6)

where $\textbf{M}_{s}$ is a binary mask to indicate the region of the object. To model the motions of the object $\textbf{M}_{s}$ in the scene, we construct separate viewpoints, including camera motion $\bm{\mu}_{t}^{sce}$ and object motion $\bm{\mu}_{t}^{obj}$. We then obtain the rendered scene novel view $\textbf{I}^{sce}_{t}$ and object novel view $\textbf{I}^{obj}_{t}$ as in Equation (3). The separate optical flows, $\textbf{F}^{sce}_{s\rightarrow t}$ and $\textbf{F}^{obj}_{s\rightarrow t}$ can also be obtained as in Equation (5). The optical flows in $\textbf{F}_{s\rightarrow t}$ are mixed by the values in $\textbf{F}^{sce}_{s\rightarrow t}$ and $\textbf{F}^{obj}_{s\rightarrow t}$ in terms of mask $\textbf{M}_{s}$ to get the final optical flow maps containing camera motion and dynamic objects for training.

Depth-Aware Inpainting Although merged images give a realistic visual effect, depth changes caused by camera motion and object motion can also cause unnatural occlusions. To solve this problem, we use volume rendering to obtain the depth $\textbf{D}_{t}^{sce}$ of the scene novel view:

$$\textbf{D}_{t}=\sum_{n=1}^{N}\left(\textbf{d}_{n}^{\prime}\bm{\alpha}_{n}^{\prime}\prod_{m=1}^{n-1}\left(1-\bm{\alpha}_{m}^{\prime}\right)\right),$$

(7)

and the depth of the object novel view $\textbf{D}_{t}^{obj}$ can be obtained in the same way. We then utilize both depths to compute the occupation mask between the novel views:

$$\textbf{M}_{occ}=(1-\textbf{M}_{t})\odot\textbf{M}_{t}^{obj}\odot(\textbf{D}_{t}<\textbf{D}_{t}^{obj}),$$

(8)

which indicates the background areas in front of the object. Therefore, we are able to restore the coincidence area between the object and the background in the new image $\textbf{I}_{t}$.

FlyingChairs [9] and FlyingThings3D [18] are both popular synthetic datasets that train optical flow models. As a standard practice, we use “Ch” and “Th” respectively to represent the two datasets, and “Ch$\rightarrow$Th” means training first on “Ch” and fine-tuning on “Th”. By default, we use the RAFT pre-trained on “Ch$\rightarrow$Th” to be fine-tuned on the generated datasets and evaluated on labeled datasets.

COCO [27] is a collection of single still images and ground truth with labels for object detection or panoptic segmentation tasks. We sample 20k single-view still images from the train2017 split following Depthstillation [1] to generate virtual images and optical flow maps.

DAVIS [37] provides high-resolution videos and it is widely used for video object segmentation. We use all the 10581 images of the unsupervised 2019 challenge to generate datasets by MPI-Flow and other state-of-the-art optical flow dataset generation methods.

KITTI2012 [11] and KITTI2015 [34] are well-known benchmarks for optical flow estimation. There are multi-view extensions (4,000 for training and 3,989 for testing) datasets with no ground truth. We use the multi-view extension images (training and testing) of KITTI 2015 to generate datasets, separately. By default, we evaluate the trained models on KITTI 12 training set and KITTI 15 training set in the tables following [1] and [12], abbreviated as “KITTI 15” and “KITTI 12” in the following tables.

Sintel [5] is derived from the open-source 3D animated short film Sintel. The dataset has 23 different scenes. The stereo images are RGB, while the disparity is grayscale. Although not a real-world dataset, we use it to verify the model’s generalization across domains.

Optical Flow networks. To evaluate how effective our generated data are at training optical flow models, we select RAFT [45], which represents state-of-the-art architecture for supervised optical flow and has excellent generalization capability. By default, we train RAFT on generated data for 200K steps with a learning rate of $1\times 10^{-4}$ and weight decay of $1\times 10^{-5}$, batch size of $6$, and $288\times 960$ image crops. This configuration is the default setting of RAFT fitting on KITTI with two GPUs but four times the number of training steps, following [12]. For the rest of the setup, we use the official implementation of RAFT without any modifications. All evaluations are performed on a single NVIDIA GeForce RTX 3090 GPU ¹¹1RAFT with the same parameters loaded on different GPUs yield slightly different evaluation results. Therefore, to ensure fair comparison, we download the official model weights with the best performance provided by the compared methods and evaluate them on the same GPU..

Virtual Camera Motion. To generate the novel view images from COCO and DAVIS, we adopt the same settings in [1] to build the virtual camera. For KITTI, we empirically build the camera motion with three scalars where $t_{x},t_{y}$ are in $[-0.2,0.2]$ and $t_{z}$ are in $[0.1,0.35]$. We build the camera rotation with three Euler angles $a_{x},a_{y},a_{z}$ in $[-\frac{\pi}{90},\frac{\pi}{90}]$. We use single camera motion for each image from COCO but multiple camera motions ($\times 4$ by default) from DAVIS and KITTI as in [12], due to the small number of images and the homogeneity of the scene in video data. We show how the number of camera motions impacts the optical flow network performance in the discussion.

Evaluation Metrics. We report evaluation results on the average End-Point Error (EPE) and two error rates, respectively the percentage of pixels with an absolute error greater than 3 ($>$ 3) or both absolute and relative errors greater than 3 and 5% respectively (Fl) on all pixels.

Comparison with Dataset Generation Methods. As a method for generating datasets from real-world images, we compare MPI-Flow with Depthstillation [1] and RealFlow [12], which are representative works in real-world dataset generation. In order to evaluate the effectiveness of MPI-Flow, we follow the procedures of Depthstillation and RealFlow to construct training sets from four different datasets, including COCO, DAVIS, KITTI 15 multi-view train set, and KITTI 15 multi-view test set. To ensure fair comparisons, we conduct experiments with a similar number of training sets as our competitors. Specifically, for DAVIS and KITTI, we generate four motions for each image to match RealFlow, which trains RAFT for four EM iterations with four times the amount of data. For COCO, we follow the exact same setup as Depthstillation. Since Depthstillation does not provide details for KITTI 15 train, we use its default settings to obtain the results. Trained models are evaluated on the training sets of Sintel, KITTI 12, and KITTI 15. We report the evaluation results of models with the best performance in the paper for Depthstillation and RealFlow. Furthermore, we conduct cross-dataset experiments where RAFT is trained with one generated dataset and evaluated with another.

Comparison with Unsupervised Methods. Another way to utilize real-world data is through unsupervised learning, which learns optical flow pixels directly without the need for optical flow labels. In order to further demonstrate the effectiveness of MPI-Flow, we compare our method with the existing literature on unsupervised methods. The results of this comparison can be seen in Table 3. All methods are evaluated under the condition that only images from the evaluation set could be used, without access to ground truth labels. For evaluation, we train the RAFT on our generated dataset with images from the KITTI 15 training set. Our MPI-Flow outperform all unsupervised methods in terms of EPE on both the KITTI 12 training set and KITTI 15 training set, with no need for any unsupervised constraints. However, our method performs better on EPE but has slightly lower Fl than SMURF, mainly because SMURF employs multiple frames for training.

Comparison with Supervised Methods. To further prove the effectiveness of MPI-Flow, we use KITTI 15 train set to fine-tune RAFT pre-trained by our generated dataset with images from KITTI 15 test set. The evaluation results on KITTI 15 train and KITTI 15 test are shown in Table 5. We achieve state-of-art performance on KITTI 2015 test benchmark compared to supervised methods on training RAFT.

Qualitative Results. Figure 4 show the generated images from the methods utilizing real-world images, as presented in Table 1. In this comparison, we use images from the KITTI 15 dataset as source image input. The images generated by RealFlow [12] and Depthstillation [1] with artifacts degrade the image realism. In contrast, MPI-Flow generates more realistic images than the other two methods. More results are shown in Figures 5 and 6.

Object Motion and Depth-Aware Inpainting. We conduct a series of ablation studies to analyze the impact of different choices in our proposed MPI-Flow for new image synthesis, including object motion, depth-aware inpainting, and multiple objects. “Multiple objects” indicates multiple moving objects in each image. To measure the impact of these factors, we generate new images from the KITTI 15 training set to train RAFT and evaluate them on the KITTI 12 training set and KITTI 15 training set. Because there are multiple combinations of these factors, we test by incrementally adding components of our approach, as shown in Table 4. In the first row, we show the performance achieved by generating new images without dynamic objects, thus assuming that optical flow comes from camera motion only. Then we incrementally add single-object motion, depth-aware inpainting, and multi-object motion to model a more realistic scenario. There are considerable improvements in almost all datasets on various metrics, except for the KITTI 12 training set, possibly due to the infrequent dynamic object motion in this dataset. The EPE on KITTI 15 remains relatively stable within the normal margin after adding the depth-aware inpainting module and the multi-object trick.

Camera Motion Parameters. We also conduct a series of ablation studies to analyze the impact of different parameters on camera motions. We first show the effect of $t_{z}$, which indicates the range of moving forward and backward distances, as shown in Table 4. We set the minimum $t_{z}$ to $0.1$ by default, considering that most cameras on vehicles only move forward in the KITTI dataset. We also show the effect of $t_{x}$ and $t_{y}$, representing the distance from left to right and from up to down, respectively. Because there are multiple combinations of parameters, we only test a specific parameter of our method in isolation. Settings used by default are underlined. The results show that a more reasonable range of camera motion leads to better performance.

Model Architectures The default model used in our paper is RFAT [45]. Table 10 shows that our proposed MPI-Flow also works for PWC-Net [44] compared with depthstillation [1] in a fair setting. We generate data from COCO and DAVIS and train PWC-Net, in which the results are still better than PWC-Net trained on Ch$\rightarrow$Th or datasets generated from depthstillation.

Amount of Virtual Motions. We can generate multiple camera motions with multiple dynamic objects for any given single image and thus a variety of paired images and ground-truth optical flow maps. Thus we can increase the number of camera motions to generate more data. Table 6 shows the impact of different amounts of camera motions on model performance. Interestingly, MPI-Flow with $4$ motions per image already allows for strong generalization to real domains, outperforming the results achieved using synthetic datasets shown in the previous evaluation results. Increasing the motions per image by factors $10$ and $20$ both lead to better performance on KITTI 12 and KITTI 15 compared to $4$ and $1$. Using $40$ motions per image gives the best performance on KITTI 15 in terms of Fl. It indicates that a more variegate image content in the generated dataset may be beneficial for generalization to real applications. Table 7 shows the effect of amounts of dynamic objects on model performance. Increasing the number of dynamic objects improves the performance of the model on KITTI 12 but slightly decreases it on KITTI 15 in terms of EPE.

Quantity of Source Images The number of source images affects the scene diversity of the generated dataset. Empirically, more source images will be more conducive to model learning, as verified in Table 11. We verify the effect of the number of source images on the MF-COCO. The model performance is significantly improved by increasing the number of source images.