Semi-supervised Deep Large-baseline Homography Estimation with Progressive Equivalence Constraint

Traditional homography estimation methods usually combine classic or learning-based feature extraction and matching algorithms such as SIFT (Lowe 2004), ORB (Rublee et al. 2011), BEBLID (Suárez et al. 2020), SuperPoint (DeTone, Malisiewicz, and Rabinovich 2018), SOSNet (Tian et al. 2019), SuperGlue (Sarlin et al. 2020), LoFTR (Sun et al. 2021), and subsequently solve direct linear transform with outliner suppression such as RANSAC (Fischler and Bolles 1981), MAGSAC (Barath, Matas, and Noskova 2019). However, feature-based methods usually crash in challenging scenes where sufficient feature matches cannot be obtained. In addition, some methods can also solve a homography directly by using the Lucas-Kanade algorithm (Baker and Matthews 2004) or calculating the sum of squared differences (SSD) between two images without extracted feature matches. A randomly initialized homography is optimized in this way iteratively.

Following the development of learning-based image alignment methods, such as optical flow (Li, Luo, and Liu 2021; Luo et al. 2021; Han et al. 2022) and dense correspondence (Truong et al. 2021a; Liu et al. 2021a), a deep homography estimation network was first proposed by (DeTone, Malisiewicz, and Rabinovich 2016). Deep homography estimation methods can be divided into two categories: supervised and unsupervised. The former ones (Shao et al. 2021; Cao et al. 2022) utilize the generated image pairs with ground-truth labels to train their models, but their generalization ability is limited due to the lack of realistic scene parallax in synthetic images. Unsupervised methods (Nguyen et al. 2018; Kharismawati et al. 2020; Ye et al. 2021) optimize their models with real-world image pairs by minimizing the photometric distance from the source image warped by the estimated homography to the target image. To be more robust, some methods (Zhang et al. 2020; Hong et al. 2022) introduce efficient masks to replace classic outlier rejection methods to remove undesired regions or focus on the dominant plane. However, most of the previous methods are proposed to estimate the homography of image pairs with a small-baseline, large-baseline homography estimation, a field with broader applications, has long been ignored.

Recently, some image stitching methods (Nie et al. 2021, 2020) use an individual homography estimation network for coarse alignment and optimize the pre-aligned images by reconstruction networks to achieve better stitching results in large-baseline scenes. However, their networks are not specially designed for such tasks, leading to unsatisfactory results. In addition, some geometric matching methods (Nie et al. 2022a; Truong, Danelljan, and Timofte 2020; Truong et al. 2021b) can also be applied to solve the large-baseline image alignment. But their alignment is mainly realized by mesh flow or dense flow, which contains the local motion information of the images, while the homography matrix only represents the global motion. In contrast, we propose a progressive estimation strategy and a semi-supervised consistency constraint without photometric loss to address the large-baseline homography estimation.

In this section, we introduce a progressive strategy to address large-baseline homography estimation. Specifically, we transform the large-baseline into several intermediate stages by inserting some intermediate images, i.e., $I_{s_{i}}\in\mathbb{R}^{H\times W\times 3}$ ($i\in[0,n]$), into the source image $I_{s}\in\mathbb{R}^{H\times W\times 3}$ and target image $I_{t}\in\mathbb{R}^{H\times W\times 3}$. We generate the intermediate ones by a set of pre-defined homographies, i.e., $I_{s_{i+1}}=\mathcal{W}_{i+1}(I_{s_{i}},\textbf{H}_{gt}^{i+1})$, where $\mathcal{W}_{i+1}$ represents the warp operation by the $\textbf{H}_{gt}^{i+1}$ and $I_{s_{0}}$ is the initial $I_{s}$. By randomly sampling the pre-defined homographies with non-identity matrices, it is effective to avoid the degenerate solutions of accumulative multiplication results of intermediate homographies (Truong et al. 2021b). In addition, we ensure that the non-overlap rate of the two inserted images is smaller than that of the source and target image.

After generating $n$ intermediate images, we get $n+2$ sets of image pairs, i.e., $(I_{s_{i}},I_{s_{i+1}})$, $(I_{s_{n}},I_{t})$, and $(I_{s},I_{t})$. Our goal is to train a neural network $f_{\theta}$, with parameters $\theta$, that predicts homography matrix $\textbf{H}_{s_{i}s_{i+1}}=f_{\theta}(I_{s_{i}},I_{s_{i+1}})$, $\textbf{H}_{s_{n}t}=f_{\theta}(I_{s_{n}},I_{t})$, $\textbf{H}_{st}=f_{\theta}(I_{s},I_{t})$ relating $I_{s_{i}}$ to $I_{s_{i+1}}$, $I_{s_{n}}$ to $I_{t}$, and $I_{s}$ to $I_{t}$ respectively. Multiplying all of the intermediate homographies $\textbf{H}_{s_{i}s_{i+1}}$ and the $\textbf{H}_{s_{n}t}$ should be equal to the $\textbf{H}_{st}$. With this equivalence constraint, we can enforce the network to optimize the homographies themselves. To achieve this, we propose a semi-supervised homography identity loss to train our network, which is described in the following section.

The point-wise correspondence between each set of image pairs can be the mapping of the corresponding homography matrix. Let us denote $\textbf{X}_{s_{i}}$ and $\textbf{X}_{t}$ as the meshgrid coordinate sets of $I_{s_{i}}$ and $I_{t}$, respectively. The point-wise correspondence of $\textbf{X}_{s_{i}}$ and $\textbf{X}_{s_{i+1}}$ is associated by the $\textbf{H}_{s_{i}s_{i+1}}$ using $\textbf{X}_{s_{i+1}}=\textbf{H}_{s_{i}s_{i+1}}\textbf{X}_{s_{i}}$. Likewise, the correspondence of $\textbf{X}_{s_{n}}$ and $\textbf{X}_{t}$ is related by the $\textbf{H}_{s_{n}t}$. Accordingly, the coordinate correspondence between the source and target image can be expressed as $\textbf{X}_{t}=\textbf{H}_{st}\textbf{X}_{s}=(\textbf{H}_{s_{n}t}\times\textbf{H}_{s_{n-1}s_{n}}\times\cdots\times\textbf{H}_{s_{0}s_{1}})\textbf{X}_{s}$, where $\times$ denotes the cross-product operation. Therefore, the homography of the source image and the target image $\textbf{H}_{st}$ can be obtained by multiplying the $\textbf{H}_{s_{n}t}$ and $\textbf{H}_{s_{i}s_{i+1}}$. The corresponding identity equation can be expressed as

$$\prod\limits_{i=n-1}^{0}\textbf{H}_{s_{i}s_{i+1}}=\textbf{H}_{s_{n}t}^{-1}\times\textbf{H}_{st}.$$

(1)

In essence, our optimization goal is not to minimize the distance between the warped source image and the target image, but to minimize the error between estimated homographies based on Eq.(1) in an unsupervised manner. The unsupervised objective function ($\mathcal{L}_{unsup}$) is formulated as

$$\mathcal{L}_{unsup}=|\textbf{H}_{s_{n}t}^{-1}\times\textbf{H}_{st}-\prod\limits_{i=n-1}^{0}\textbf{H}_{s_{i}s_{i+1}}|_{1},$$

(2)

where $|\cdot|_{1}$ denotes the L1 norm. Since the intermediate images are generated through the pre-defined homographies, so we can estimate the homographies of intermediate images in a supervised way. The supervised objective function ($\mathcal{L}_{sup}$) is formulated as

$$\mathcal{L}_{sup}=\sum\limits_{i=0}^{n-1}|\textbf{H}_{s_{i}s_{i+1}}-\textbf{H}_{gt}^{i+1}|_{1},$$

(3)

and the $\textbf{H}_{s_{i}s_{i+1}}$ in Eq.(2) can be replaced with the $\textbf{H}_{gt}^{i+1}$. However, due to the cancellation effect amid the estimated homography terms, artlessly replacing $\textbf{H}_{s_{i}s_{i+1}}$ with $\textbf{H}_{gt}^{i+1}$ may obtain degeneration solutions, e.g., $\textbf{H}_{st}=I_{3\times 3}$. To avoid this situation, we rewrite Eq.(2) as

$$\mathcal{L}_{unsup}=\sum\limits_{i=0}^{n-1}\lambda_{i}|(\textbf{H}_{s_{i+1}t}^{-1}\times\textbf{H}_{st})-\prod\limits_{j=i}^{0}\textbf{H}_{gt}^{j+1}|_{1},$$

(4)

the detailed derivation is conducted in the supplementary material.

Our final semi-supervised homography identity loss $\mathcal{L}_{HIL}$ combines the $\mathcal{L}_{sup}$ and $\mathcal{L}_{unsup}$ as $\mathcal{L}_{HIL}=\mathcal{L}_{unsup}+\lambda_{w}\mathcal{L}_{sup}$, where the $\lambda_{w}$ is a weighting factor, we eliminate this hyper-parameter by automatically balancing the weights over each training batch as $\lambda_{w}=\mathcal{L}_{unsup}/\mathcal{L}_{sup}$.

There are two challenges for the large-baseline homography estimation: 1) the overlap rate between two images is low, and 2) the receptive field of CNN-based models is limited (Nie et al. 2022b). To overcome these problems, we design a multi-scale homography estimation network, as shown in Fig. 3, that combines a multi-scale CNN encoder and correlation layers to leverage feature information and expand the receptive field. Subsequently, the global and local correlation maps are fed into a coarse-to-fine homography estimation module to obtain the final results.

For the rest feature maps, i.e., $\mathcal{F}^{0}$, $\mathcal{F}^{1}$, $\hat{\mathcal{F}}^{1}$, we apply the local correlation layer proposed in (Truong et al. 2020) to evaluate the feature similarity between two feature maps, denoted as $\textbf{C}_{l}^{0}$, $\textbf{C}_{l}^{1}$, and $\hat{\textbf{C}}_{l}^{1}$. The search region $R$ is set to constrain the search space and result in local correlation maps with the size of $\textbf{C}_{l}$ is $\frac{H^{(^{\prime})}}{2^{2+k}}\times\frac{W^{(^{\prime})}}{2^{2+k}}\times(2R+1)^{2}$.

As mentioned in (DeTone, Malisiewicz, and Rabinovich 2016), it is non-trivial to directly estimate a homography matrix. Therefore, we use the homography flow as the supervision object during the training stage. Besides, we resize the initial images to fixed-resolution to improve the applicability to high resolution images, our supervised and unsupervised objectives are accordingly converted into

$$\mathcal{L}_{sup}=\sum\limits_{i=0}^{n-1}|\textbf{F}_{s_{i}s_{i+1}}-\textbf{F}_{gt}^{i+1}|_{1}+\sum\limits_{i=0}^{n-1}|\hat{\textbf{F}}_{s_{i}s_{i+1}}-\hat{\textbf{F}}_{gt}^{i+1}|_{1},$$

(6)

	$\displaystyle\mathcal{L}_{unsup}=\sum\limits_{i=0}^{n-1}\lambda_{i}$	$\displaystyle(|\textbf{F}_{st}-\textbf{F}_{s_{i+1}t}-\sum\limits_{j=i}^{0}\textbf{F}_{gt}^{j+1}|_{1}$		(7)
		$\displaystyle+|\hat{\textbf{F}}_{st}-\hat{\textbf{F}}_{s_{i+1}t}-\sum\limits_{j=i}^{0}\hat{\textbf{F}}_{gt}^{j+1}|_{1}).$

The most relative representations of our homography flow are the 8 bases flow (Ye et al. 2021) and optical flow. While the former performs well in small-baseline scenes, it crashes in large-baselines. Compared to optical flow, with the assistance of our supervised objective and LRR blocks, our homography flows tend to represent the global motion between two images. Importantly, the homography flows are only used to facilitate training, and $\mathcal{DLT}$ will be used to transform the flows into homography matrices.

As shown in Table 1, our method achieves state-of-the-art performance in all categories of the large-baseline dataset. In the regular (RE-L) scenes, our method and SIFT+RANSAC achieve the best results, because the feature-based methods can obtain sufficient high-quality matching points in regular scenes and thus perform well. But feature-based methods fail in other challenging scenes, especially in low-light (LL-L) and low-texture (LT-L), while our method does not rely on feature detection and correspondence matching, being more robust than traditional methods in these scenarios. For example, our method reduces the error on LT-L by 66.44$\%$ compared to SIFT+MAGSAC. Moreover, image alignment methods perform better than feature-based methods in some challenging scenarios, our method still produces lower errors, e.g., our method reduces the error on LL-L by 53.03$\%$ compared to UNSUPDIS. The small-foreground (SF-L) and large-foreground (LF-L) scenes contain dynamic objects, affecting the estimation of homography. Compared with other deep learning-based methods using photometric losses for optimization, our method outperforms them in LF-L and SF-L benefiting from our homography identity loss.

Additionally, we also conduct experiments on the small-baseline dataset (Zhang et al. 2020), which also contains 5 categories RE-S, LT-S, LL-S, LF-S, and SF-S. Considering the small relative motion between two images in small-baseline scenes, we set $n=1$ and reduce the non-overlap rate of the source and inserted image. As reported in Table 2, our method outperforms the existing four deep learning-based methods with the error reduced from 0.50 to 0.44.

We conduct extensive ablation studies to verify the effectiveness of our proposed components, and the results are reported in Table 3.

Although our method achieves state-of-the-art performance in large-baseline scenes compared with the existing methods, it still has its limitation of being applied to scenes with multiple planes where a homography theoretically cannot perform alignment well. We will leave the solution for the multiple planes alignment as future work.