LiteFlowNet3: Resolving Correspondence Ambiguity for More Accurate Optical Flow Estimation

We first provide a concise description on the construction of cost volume in optical flow CNNs. Suppose a pair of images $I_{1}$ (at time $t=1$) and $I_{2}$ (at time $t=2$) is given. We convert $I_{1}$ and $I_{2}$ respectively into pyramidal feature maps ${\mathcal{F}}_{1}$ and ${\mathcal{F}}_{2}$ through a feature encoder. We denote ${\bf x}$ as a point in the rectangular domain $\Omega\subset~{}\mathbb{R}^{2}$. Correspondence between $I_{1}$ and $I_{2}$ is established by computing the dot product between two high-level feature vectors in the individual feature maps ${\mathcal{F}}_{1}$ and ${\mathcal{F}}_{2}$ as follows [6]:

where $D$ is the maximum matching radius, $c({\bf x};D)$ (a 3D column vector with length $2D+1$) is the collection of matching costs between feature vectors ${\mathcal{F}}_{1}({\bf x})$ and ${\mathcal{F}}_{2}({\bf x}^{\prime})$ for all possible ${\bf x}^{\prime}$ such that $\|{\bf x}-{\bf x}^{\prime}\|_{\infty}=D$, and $N$ is the length of the feature vector. Cost volume $C$ is constructed by aggregating all $c({\bf x};D)$ into a 3D grid. Flow decoding is then performed on $C$ using convolutions (or native winner-takes-all approach [17]). The resulting flow field ${\bf u}:\Omega\rightarrow\mathbb{R}^{2}$ provides the dense correspondence from $I_{1}$ to $I_{2}$. In the following, we will omit variable $D$ that indicates the maximum matching radius for brevity and use $c({\bf x})$ to represent the cost vector at x. When we discuss operations in a pyramid level, the same operations are applicable to other levels.

Given a pair of images, the existence of partial occlusion and homogeneous regions makes the establishment of correspondence very challenging. This situation also occurs on feature space because simply transforming images into feature maps does not resolve the correspondence ambiguity. In this way, a cost volume is corrupted and the subsequent flow decoding is seriously affected. Conventional methods [26, 30] address the above problem by filtering a cost volume prior to the decoding. But there has not been any existing works to address this problem for optical flow CNNs. Some studies [2, 10, 12] have revealed that applying feature-driven convolutions on feature space is an effective approach to influence the feed-forward behavior of a network since the filter weights are adaptively constructed. Therefore, we devise to filter outliers in a cost volume by using an adaptive modulation. We will show that our modulation approach is not only effective in improving the flow accuracy but also parameter-efficient.

An overview of cost volume modulation is illustrated in Fig. 2b. At a pyramid level, each cost vector $c({\bf x})$ in cost volume $C$ is adaptively modulated by an affine transformation ($\alpha({\bf x}),\beta({\bf x})$) as follows:

where $c_{m}({\bf x})$ is the modulated cost vector, “$\otimes$” and “$\oplus$” denote element-wise multiplication and addition, respectively. The dimension of the modulated cost volume is same as the original. This property allows cost volume modulation to be jointly used and trained with an existing network without major changes made to the original network architecture.

To have an efficient computation, the affine parameters $\{\alpha({\bf x}),\beta({\bf x})\},\forall{\bf x}\in\Omega$, are generated altogether in the form of modulation tensors ($\alpha,\beta$) having the same dimension as $C$. As shown in Fig. 2a, we use cost volume $C$, feature ${\mathcal{F}}_{1}$ from the encoder, and confidence map $M$ at the same pyramid level as the inputs to the modulation parameter generator. The confidence map is introduced to facilitate the generation of modulation parameters. Specifically, $M({\bf x})$ pinpoints the probability of having an accurate optical flow at ${\bf x}$ in the associated flow field. The confidence map is constructed by introducing an additional output in the preceding optical flow decoder. A sigmoid function is used to constraint its values to $[0,1]$. We train the confidence map using a L2 loss with the ground-truth label $M_{gt}({\bf x})$ as follows:

where ${\bf u}_{gt}({\bf x})$ is the ground truth of ${\bf u}({\bf x})$. An example of predicted confidence maps will be provided in Section 4.2.

In coarse-to-fine flow estimation, a flow estimate from the preceding decoder is used as a flow initialization for the subsequent decoder. This highly demands the previous estimate to be accurate. Otherwise, erroneous optical flow is propagated to subsequent levels and affects the flow inference. Using cost volume modulation alone is not able to address this problem. We explore local flow consistency [29, 33] and propose to use a meta-warping for improving the flow accuracy.

Intuitively, we refine a given flow field by replacing each inaccurate optical flow with an accurate one from a nearby position using the principle of local flow consistency. As shown in Fig. 4, suppose an optical flow ${\bf u}({\bf x}_{1})$ is inaccurate. With some prior knowledge, 1) a nearby optical flow ${\bf u}({\bf x}_{2})$ such that ${\bf x}_{2}={\bf x}_{1}+{\bf d}({\bf x}_{1})$ is known to be accurate as indicated by a confidence map; 2) the pyramidal features of $I_{1}$ at ${\bf x}_{1}$ and ${\bf x}_{2}$ are similar i.e., $\mathcal{F}_{1}({\bf x}_{1})\sim\mathcal{F}_{1}({\bf x}_{2})$ as indicated by an auto-correlation cost volume. Since image points that have similar feature vectors have similar optical flow in a neighborhood, we replace ${\bf u}({\bf x}_{1})$ with a clone of ${\bf u}({\bf x}_{2})$.

The previous analysis is just for a single flow vector. To cover the whole flow field, we need to find a displacement vector for every position in the flow field. In other words, we need to have a displacement field for guiding the meta-warping of flow field. We use a warping mechanism that is similar to image [13] and feature warpings [10, 27]. The differences are that our meta-warping is limited to two channels and the physical meaning of the introduced displacement field no longer represents correspondence across images.

An overview of flow field deformation is illustrated in Fig. 5b. At a pyramid level, we replace ${\bf u}({\bf x})$ with an neighboring optical flow by warping of ${\bf u}({\bf x})$ in accordance to the computed displacement ${\bf d}({\bf x})$ as follows:

In particular, not every optical flow needs an amendment. Suppose ${\bf u}({\bf x_{0}})$ is very accurate, then no flow warping is required i.e., ${\bf d}({\bf x_{0}})\sim{\bf 0}$.

To generate the displacement field, the location of image point having similar feature as the targeted image point needs to be found. This is accomplished by decoding from an auto-correlation cost volume. The procedure is similar to flow decoding from a normal cost volume [6]. As shown in Fig. 5a, we first measure the feature similarity of targeted point at ${\bf x}$ and its surrounding points at ${\bf x}^{\prime}$ by computing auto-correlation cost vector $c_{a}({\bf x};D)$ between features $\mathcal{F}_{1}({\bf x})$ and $\mathcal{F}_{1}({\bf x}^{\prime})$ as follows:

where $D$ is the maximum matching radius, ${\bf x}$ and ${\bf x}^{\prime}$ are constrained by $\|{\bf x}-{\bf x}^{\prime}\|_{\infty}=D$, and $N$ is the length of the feature vector. The above equation is identical to Eq. (1) except using features from $I_{1}$ only. Auto-correlation cost volume $C_{a}$ is then built by aggregating all cost vectors into a 3D grid.

To avoid trivial solution, confidence map $M$ associated with flow field ${\bf u}$ that is constructed by the preceding flow decoder (same as the one presented in Section 3.2) is used to guide the decoding of displacement from $C_{a}$. As shown in Fig. 5a, we use cost volume $C_{a}$ for the auto-correlation of ${\mathcal{F}}_{1}$ and confidence map $M$ at the same pyramid level as the inputs to the displacement field generator. Rather than flow decoding from the cost volume as the normal descriptor matching [6], our displacement decoding is performed on the auto-correlation cost volume and is guided by the confidence map.

We evaluate LiteFlowNet3 on the popular optical flow benchmarks including Sintel clean and final passes [4], KITTI 2012 [7], and KITTI 2015 [22]. We report average end-point error (AEE) for all the benchmarks unless otherwise explicitly specified. More results are available in the supplementary material [9].

To study the role of each proposed component in LiteFlowNet3, we disable some of the components and train the resulting variants on FlyingChairs. The evaluation results on the public benchmarks are summarized in Table 4 and examples of flow fields are illustrated in Fig. 8.