Practical Stereo Matching via Cascaded Recurrent Network with Adaptive Correlation

We observe that it is difficult to implement perfect calibration for real-world stereo cameras. For instance, the two cameras may not be strictly placed on the horizontal epipolar line, resulting in slight rotations in 3D space; or images from camera lenses usually have residual distortion even after they are rectified. As a result, for a stereo image pair, the corresponding points may not locate on the same scan-line. We thus propose an Adaptive Group Correlation Layer (AGCL) to reduce the matching ambiguity in this situation, achieving better performance compared to all-pairs matching [45, 23] while only local correlation is computed.

Local Feature Attention. Instead of computing global correlation for every pair of pixels, we only match points in a local window to avoid large memory consumption and computation cost. In light of LoFTR [41] for sparse feature matching, we add an attention module before correlation computation in the first stage of cascades in order to aggregate global context information in single or cross feature maps. Following [41], we add positional encoding to the backbone output, which enhances positional dependence of the feature maps. The self and cross attention is computed alternately, where a linear attention layer is used to reduce computation complexity.

2D-1D Alternate Local Search. Different from the flow estimation network RAFT [45] and its stereo version [23] where all-pairs correlation is computed by a matrix multiplication of two $C\times H\times W$ feature maps, which outputs a 4D $H\times W\times H\times W$ or 3D $H\times W\times W$ cost volume, we only compute correlation in a local search window that outputs a much smaller volume of $H\times W\times D$ to save the memory and computation cost. $H$ and $W$ denote the height and width of the feature maps, and $D$ is the number of correlation pairs much smaller than $W$. Our correlation computation is also distinct from cost volume based stereo networks like [7, 18, 51, 49] where the search range is related to the maximum displacement of foreground objects. This fixed range is much larger than the number of local correlation pairs we use, which leads to more noisy interference. Furthermore, we don’t need to preset the range when the model generalizes to stereo pairs with different baselines.

Given two resampled and attended feature maps $\mathbf{F}_{1}$ and $\mathbf{F}_{2}$, the local correlation at position $(x,y)$ can be denoted as

where $x^{\prime}=x+f(d)$, $y^{\prime}=y+g(d)$, $\mathrm{Corr}(x,y,d)\in\mathbb{R}^{H\times W\times D}$ is the matching cost of $d$-th ($d\in[0,D-1]$) correlation pair, $C$ is the number of feature channels, $f(d)$ and $g(d)$ denote the fixed offset of current pixels in horizontal and vertical directions.

Traditionally, search direction between two rectified images only lies on the epipolar line in stereo matching. To deal with non-ideal stereo rectification cases, we adopt a 2D-1D alternate local search strategy to improve the matching accuracy. In 1D search mode, we set $g(d)=0$ and $f(d)\in[-r,r]$, where $r=4$. Positive displacement value of $f(d)$ is reserved to adjust inaccurate results after every iterative sampling. The results computed by Eq. 1 are stacked and concatenated at channel dimension for the final correlation volume. In 2D search mode, a $k\times k$ grid with dilation $l$ similar to dilated convolution[53] is used for correlation computation. We set $k=\sqrt{2r+1}$ to make sure the output features have the same number of channels so that they can be fed to a shared-weight update block. Cooperating with iterative resampling, alternate local search also acts as a propagation module for recurrent refinement, where the network learns to replace the biased prediction on current location with its more accurate neighbors.

Deformable search window. Stereo matching often suffers from ambiguity in occlusion or textureless areas. Correlation computed in a fixed-shape local search window tends to be vulnerable to those cases. Extending deformable convolution[57] to correlation computation, we use a content adaptive search window for correlation pairs generation, which is different from AANet[49] where a similar strategy is adopted only in cost aggregation. With the learned additional offset $\mathrm{d}x$ and $\mathrm{d}y$, the new correlation can be computed as

where $x^{\prime\prime}=x+f(d)+\mathrm{d}x$, $y^{\prime\prime}=y+g(d)+\mathrm{d}y$. Fig. 4 shows how offsets change the formation of a conventional search window.

Group-wise correlation. Inspired by [12] which introduces the group-wise 4D cost volume, we split the feature map into $\mathcal{G}$ groups to compute local correlation individually. Finally, we concatenate $\mathcal{G}$ correlation volumes of $D\times H\times W$ in channel dimension to get the $\mathcal{G}D\times H\times W$ output volume. The procedure is shown in Fig. 3.

For non-texture or repetitive-texture areas, matching is more robust using low-res and high-level feature maps due to large receptive field and sufficient semantic information. However the details of fine structures may be lost in such feature maps. In order to maintain robustness and preserve the details in high-res input simultaneously, we propose cascaded recurrent refinement for correlation computing and disparity updating.

Recurrent Update Module. We build a Recurrent Update Module (RUM) based on GRU blocks and our Adaptive Group Correlation Layer (AGCL). Unlike in RAFT[45] where the feature pyramid is constructed in a single correlation layer with the output being merged into one volume, we compute correlations for every feature map respectively in different cascade levels and refine the disparities for several iterations independently. As is shown in Fig. 3, the “sampler” samples locations of grouped feature taking coordinate grid derived from $f_{n}$ as input. $\{f_{1},...,f_{n}\}$ are intermediate predictions of $n$ iterations with initialization $f_{0}$. Current correlation volume is constructed with learned offsets $o\in\mathbb{R}^{2\times(2r+1)\times h\times w}$. The GRU blocks update current prediction and feed it to the AGCL in next iteration.

Cascaded Refinement. Except for the first level of cascades, which starts at 1/16 of the input resolution with disparity initialized to all zeros, other levels take the upsampled version of prediction from previous level as initialization. Though handling different levels of refinement, all RUMs share the same weights. After the last refinement level, convex upsampling [45] is conducted to get the final prediction at input resolution.

As is discussed in previous sections, during training we use a three-level feature pyramid at fixed resolutions to do hierarchical refinement. However, for images of higher resolution as input, more downsampling should be done in order to enlarge the receptive field for extracted features and correlation computation. But for small objects with large displacement in high resolution images, features in these regions may suffer from deterioration with direct downsampling. To solve this problem, we designed a stacked cascaded architecture with shortcuts for inference. Specifically, we downsample the image pair in advance constructing an image pyramid and feed them into the same trained feature extraction network to take advantage of multi-level context. An overview of the stacked cascade architecture is shown on the right of Fig. 2, where skip connections in the same stage are not displayed for brevity. For a specific stage of the stacked cascades (denoted as rows in Fig. 2), all the RUMs in that stage will be used together with the last RUM in the stage of higher resolution. All stages of the stacked cascades share the same weight during training, so no fine-tuning is needed.

For each stage $s\in\{\frac{1}{16},\frac{1}{8},\frac{1}{4}\}$ of our feature pyramid, we resize the sequence of output $\{\mathbf{f}_{i}^{s},\cdots,\mathbf{f}_{n}^{s}\}$ to the full prediction resolution with the upsampling operator $\mu_{s}$, and use the exponentially weighted $l_{1}$ distance similar to RAFT [45] as the loss function (with $\gamma$ set to 0.9). Given ground truth disparity $\mathbf{d}_{\mathrm{gt}}$, the total loss is defined as:

Compared to previous synthetic datasets, our data generating pipeline devotes extra attention to challenging cases in real-world scenes, and features various enhancements. We make use of Blender [3] to generate our synthetic training data. Each scene consists of left-right image pairs and the corresponding pixel-accurate dense disparity map, captured with dual virtual cameras and customarily positioned objects. Our major design considerations are described as below, with some examples shown in Fig. 5.

Shape. We diversify the shapes of the models used as the main scene content with multiple sources: 1) The ShapeNet [6] dataset with over 40,000 3D models of common objects with varied shapes, forming our basic source of content. 2) Blender’s sapling tree gen add-on, providing fine-detailed and cluttered disparity maps. 3) We use blender’s internal basic shapes combined with the wireframe modifier to generate models for challenging scenes featuring holes and open-work structures.

Lighting and texture. We place different types of lights with random color and luminance at random position inside the scene, resulting in a complex lighting environment. Real world images are used as textures for objects and scene background, particularly hard scenes containing repeated patterns or lacking visible features. Additionally, we exploit the light tracing ability of Blender’s Cycles renderer and randomly set objects as transparent or with metallic reflection, in order to cover real-world scenes with similar attributes.

Disparity distribution. To cover different baseline settings, we make efforts to ensure the disparity of the generated data distributes smoothly within a wide range. We put objects within a frustum-shaped space formed by the cameras’ field of view and a max distance. The exact position of each object is randomly chosen from a probability distribution, then the object is scaled according to its distance to prevent blocking the view. This practice results in a randomized but controllable disparity distribution.

We evaluate our method on three popular public benchmarks. Middlebury 2014 [33] provides 23 high-resolution image pairs under different lighting environments. Captured with large-baseline stereo cameras, the maximum disparity in Middlebury can exceed 600 pixels. ETH3D [36] consists of 27 monochrome stereo image pairs with disparity sampled by a laser scanner, covering both indoor and outdoor scenes. KITTI 2012/2015 [28] consists of 200 wide-angle stereo image pairs of street views, with lidar-sampled sparse disparity ground truth.

In addition to our rendered dataset, we collect major public datasets for training, including Sceneflow [27], Sintel [5] and Falling Things [46]. Sceneflow contains 39k training pairs of multiple synthetic scene setups. Falling things contains a large amount of images from scenes of household object models. Sintel provides 1.2k stereo pairs from various synthetic sequences. The other data sources we utilize are InStereo2K [1], Carla [9] and AirSim [37].

For evaluation, we follow the popular metrics including AvgErr (average error), Bad2.0 (percentage of pixels with disparity error larger than 2 pixels) [35, 36], D1-all (percentage of disparity outlier pixels in left image) [11], etc.

Training. Our network is implemented with Pytorch [31] framework. The model is trained on 8 NVIDIA GTX 2080Ti GPUs, with a batch size of 16. The whole training process is set to 300,000 iterations. We use the Adam [19] optimizer with a standard learning rate of 0.0004. We perform a warm-up process of 6,000 iterations at the beginning of the training where the learning rate is linearly increased from 5% to 100% of the standard value. After 180,000 iterations, the learning rate is linearly decreased down to 5% of the standard value towards the end of the training process. The model is trained with an input size of $384\times 512$. All training samples undergo a set of augmentation operations before getting fed into the model.

Augmentation. To imitate the camera module inconsistencies and non-ideal rectification, we employ multiple data augmentation techniques for training. Firstly, we apply asymmetric chromatic augmentations for the two inputs respectively, including shifts in brightness, contrast and gamma. To further enhance the robustness for rectification error in real-world images, we conduct spatial augmentation applied only to the right image: slightly random homography transformation and vertical shift at a very small range ($<$ 2 pixels). To avoid mismatching in ill-posed regions, we use random rectangle occlusion patches with height and width between 50 and 100 pixels. Finally, to fit input data from various sources into the network’s training input size, the group of stereo images and disparity undergoes random resize and crop operations.

In this section we evaluate our model on different settings to prove the effectiveness of the network components. Besides the ablation study for the stacked cascades, all evaluation resolutions are $768\times 1024$.

Correlation types. In order to compare the effect of different types of correlations, we replace our correlation layers with other forms. As shown in Tab. 1, both 2D and 1D all-pairs correlation used in [45] and [23] lead to a substantial drop in accuracy compared with their local forms. When we replace the alternate local correlation with a single 2D or 1D correlation, it harms the final precision, which is more evident when the network contains more than 1 level of cascades because the rectification error increases with the resolution.

Components in AGCL. As shown in the bottom half of Tab. 1, using a fixed correlation window without learned offsets degrades the accuracy which demonstrates the effectiveness of the adaptive mechanism. Replacing group correlation with single form and removing the local feature attention modules both deteriorate the accuracy.

Cascaded RUMs. We compare the performance of different numbers of cascade stages. As is shown in Tab. 1, using a single RUM without cascades leads to a substantial drop in precision. When changing the number of cascades, the prediction error decreases evidently when more levels of cascades are used while the correlation type keeps the same. This demonstrates the importance of our cascaded architecture.

Stacked cascades. During inference, we feed the cascades using different levels of image pyramid as input while sharing the same trained parameters. We compare the performance of different stages of cascades with various of resolutions on Middlebury. As shown in Tab. 2, the prediction error increases with the input size when only a single cascade is used. Multi-level input helps to reduce the error substantially, which demonstrates that our stacked cascades scheme enjoys a great improvement for disparity accuracy.

New synthetic data. To analyze the effectiveness of our proposed synthetic data, we sample 35,000 pairs of images from our training dataset and compare with similar-sized Sceneflow [27]. Both datasets are used to train our model with the same augmentation for 50,000 iterations. As shown in Fig. 6, our synthetic data results in lower training loss and better performance in both ETH3D and Middlebury validation data. This demonstrate that our dataset is more advantageous in domain generalization.

Middlebury. We train our network on 23 pairs of images (including 13 additional pairs with ground truth) from Middlebury 2014 dataset together with our full training set without fine-tuning. The proportion of Middlebury training set is augmented to 2% of the full training set. We evaluate the test set at $1536\times 2048$ using resized full-resolution images where 2-stage inference is adopted, and the results are submitted to the online leaderboard . We achieve the \nth1 place on the majority of the metrics among more than 120 other methods, surpassing the published state-of-the-art by 21.73% on the bad 2.0 metric, 31.00% on the A95 metric. The quantitative comparison results with other methods are shown in Tab. 3.

ETH3D. We train our network on the whole training set with a proportion of 2% augmented training data from ETH3D low-res two-view stereo dataset. Without fine-tuning, we evaluate the test set at the size of $768\times 1024$ where 2-stage inference is adopted. At the time of writing, we achieve state-of-the-art among published methods on the online benchmark for all metrics. Our method surpasses the published state-of-the-art by 59.84% on the bad 1.0 metric. Quantitative comparisons are shown in Tab. 4.

KITTI. Different from the training procedure for Middlebury and ETH3D, we fine-tune the model pre-trained on the full training set for another 50K iterations on KITTI 2012 and 2015 training sets. The initial learning rate is set to 0.0001. We augment the proportion of KITTI datasets to 75% with the rest part randomly sampled from the whole training set. During evaluation, we pad the input to $384\times 1248$ before feeding to the network and single stage inference is adopted. We achieve competitive performance on both datasets , surpassing LEAStereo [8] in KITTI 2012 by 9.47% on Out-Noc under 2 pixels error threshold. We show a visual comparison of KITTI 2015 in Fig. 8.

Compared with the real-world images from standard stereo datasets which is limited in numbers and scenes, images taken from consumer-level devices pose greater challenges to stereo matching. For fair comparisons, we trained all other stereo networks with author-released code and recommended settings on our full training set.

Holopix50K. Fig. 9 shows the qualitative comparison results of our network with several published stereo matching on Holopix50K [16] dataset in varied scenes. Pre-rectification were performed to eliminate possible negative disparity. The visual results show that our method has a significant advantage in thin objects like cat whiskers and wire meshes. We also achieve better performance on textureless areas like walls and windows.

Disturbed ETH3D. We simulate common disturbances in practical scenes on ETH3D dataset to test the robustness of our proposed method and list the quantitative results in Fig. 10. The disturbances here include image blur, color transform, chromatic noise, image perspective transform, vertical shift and spatial distortion. The results demonstrate that our method is less vulnerable to these disturbances.

Smartphone photos. Because it is difficult to obtain ground truth disparity in real-world scenes, an empirical way is to manually label a foreground mask $M_{f}$ for evaluating the disparity quality [25]. The metric of IoU (intersection over union) is commonly used in segmentation tasks. For a disparity map, we can place a threshold $t$ to obtain a foreground mask $M_{t}$, where the disparity values of the foreground are larger than $t$. The “mxIoU” means the maximum IoU between $M_{f}$ and $M_{t}$ by changing $t$. Similarly, “mxIoUbd” means mxIoU in a banded area within $p$ (we set $p=4$) pixels from the boundary of $M_{f}$. The quantitative and qualitative comparison results are shown in Tab. 5 and Fig. 11 respectively.