Cascade Network with Guided Loss and Hybrid Attention for Finding Good Correspondences

Given an image pair, we first extract local features (handcrafted descriptors such as SIFT (Lowe 2004), or deep learning based descriptors, such as Hard-Net (Mishchuk et al. 2017)) of each image and perform feature matching to establish a set of putative correspondences between them. The coordinates of each correspondence in the putative set are concatenated as the input of our network, as follows:

where $N$ is the number of putative correspondences. $(x_{1}^{i},y_{1}^{i})$ and $(x_{2}^{i},y_{2}^{i})$ are the coordinates of the two feature points of $i$-th correspondence. The coordinate of each feature point is normalized by camera intrinsics (Moo Yi et al. 2018). The network extracts a feature for each correspondence and determines the probability that a correspondence is inlier based on their features as follows:

where $\Phi(\cdot)$ is the network with trained parameters. $L$ is the logit values predicted by the network. After that, the network performs a differentiable weighted eight-point algorithm (Moo Yi et al. 2018) on the correspondence to estimate the relative pose ($E$ matrix). as follows:

$g(\cdot,\cdot)$ is the weighted eight-point algorithm, $L$ is the predicted logit value and $\hat{E}$ is the estimated $E$ matrix.

The correspondence classification in our network is a binary classification task. In general, the result is evaluated by the Fn-measure ($Fn$), which considers both precision ($P$) and recall ($R$), as follows:

When $n>1$, the Fn-measure is biased in favour of recall and otherwise in favour of precision. When adopting cross entropy loss as objective function, the loss will gradually decrease under the successive optimization. However, there is no guarantee that a drop in the loss will result in an increase of Fn-measure. Therefore, the network may not be trained towards the direction of optimizing Fn-measure. Based on this observation, we propose a hypothesis, that is, whether the relationship between the cross entropy loss and Fn-measure can be established, so that the decrease of loss will lead to the increase of Fn-measure. This relationship can be expressed in the form of differential as follows:

Specifically, the proposed Guided Loss ($l$) uses the form of IB-CE-loss as follows:

where $N_{pos}$ and $N_{neg}$ are the number of positive and negative samples. $\lambda$ and $\mu$ are the weights of positive and negative samples. $y_{i}$ and $y_{j}$ is the logit value of correspondence $i$ and $j$ respectively. Meanwhile, after forward propagation of the network, all the samples are divided into four categories, including $FP$ (false positive), $FN$ (false negative), $TP$ (true positive) and $TN$ (true negative). Suppose the number of $FN$ and $FP$ samples are $X$, $Y$ respectively, then the number of $TP$ and $TN$ can be computed as follows:

and the precision ($P$) and recall ($R$) in Fn-measure ($P$, $R$ in Eq. 4) can be computed as follows:

Thus, Fn-measure is the dependent variable of $X$ and $Y$ according to Eq. 4 and 8. We express the functional relationship between Fn-measure ($Fn$) and $X,Y$ as follows:

In order to derive the relationship between Fn-measure and the loss, we also expect to express the loss as the dependent variables of $X$ and $Y$. In the forward propagation of the network, we can calculate the average loss terms of $TP$, $TN$, $FP$ and $FN$ samples respectively, denoted as $l_{TP},l_{TN},l_{FP},l_{FN}$. Then the loss in Eq. 22 can be equivalently calculated as follows:

We compute the derivative forms of loss function ($dl$) and Fn-measure ($dFn$) by $X$ and $Y$ as follows:

where $\partial l_{X}$ and $\partial l_{Y}$ are the partial derivatives of loss with respect to $X$ and $Y$, and $\partial F_{X}$ and $\partial F_{Y}$ are the partial derivatives of Fn-measure with respect to $X$ and $Y$. Then, we can draw a sufficient condition of Eq. 5 as follows:

Algorithm. The $\partial l_{X}$ and $\partial l_{Y}$ can be computed according to Eq. 23 as follows:

Meanwhile, $\partial F_{X}$ and $\partial F_{Y}$ can also be calculated by means of numerical derivatives (step 4 in Algorithm 1 ) in the training process. Obviously, to hold Eq. 12, the weights $\lambda$ and $\mu$ should be dynamically changed during training. This also reveals the problem of IB-CE-Loss using a fixed $\lambda$ and $\mu$ during training. In order to establish a relationship between loss and Fn-measure as Eq. 5, we design a weight adjustment algorithm by making Eq. 12 hold, as follows:

Specifically, when a batch of training data is sent to the network, the first step is forward propagation. After the forward propagation, we can use algorithm 1 to get $\lambda$ and $\mu$ for making Eq. 12 hold. Then we substitute $\lambda$ and $\mu$ into Eq. 22 and perform back propagation.

The basic feature extraction block of our network is the proposed hybrid attention block (HAB). As shown in Fig. 2 (a), the input of the HAB is the feature map $f^{N\times C}$ (output of last layer or data points at layer zero), where $N$ is the number of correspondences and $C$ is the number of channels. HAB integrates Bayesian attentive context normalization (BACN), batch normalization (BN) (Ioffe and Szegedy 2015), ReLU and Channel-wise Attention (CA) operations in the structure of Res-Net (He et al. 2016). Specifically, BACN is to normalize each correspondence so that the features of correct and incorrect correspondences are distinguishable. BN is adopted to accelerate network convergence and ReLU function is utilized as an activation function. Finally, the CA operation learns the statistical information on the channel to boost the performance of the network.

The key to BACN is how to better learn the weight vector for computing weighted mean and variance. In the shallow layers of the network, it’s hard to learn appropriate weight vector because the features in these layers are less recognizable. Since Lowe Ratio (Lowe 2004), which is generated during feature matching, is proved useful for determining the confidence of each correspondence being inlier, we expect to use it to make up for the dilemma of weight learning in shallow network. However, the distribution of Lowe Ratio is quite different on different datasets, while independence and identical distribution of features is a very important assumption in neural networks (Bishop 2006). To make better use of Lowe Ratio, we first use the Bayesian Model (Bishop 2006) to convert the it into a probability value. Formally, given a pair of correspondence with Lowe Ratio $r_{i}\in\mathbb{R}^{1}$, we consider $r_{i}$ as a variable and the joint probability distribution function (PDF) can be modeled as:

where $f_{in}(r_{i})=f(r_{i}|r_{i}$ belongs to an inlier$)$, $f_{out}(r_{i})=f(r_{i}|r_{i}$ belongs to an outlier$)$, and $\alpha$ is the inlier ratio of the putative correspondence set of a specific image pair. Then, the prior probability $p_{i}(in)$ that the $i$-th correspondence belongs to inlier can be calculated as follows:

Before training, we obtain the PDF of inlier ($f_{in}$) and outlier ($f_{out}$) on the training dataset with ground-truth as empirically PDF. Then for each image pair, we estimate the inlier ratio $\alpha$ using a curve fitting method (Goshen and Shimshoni 2008). Thus we assign a prior probability to each correspondence by Eq. 16.

After obtaining the prior probability for each correspondence, it will be utilized to participate in the calculation of weight vector. The architecture of weight learning is inspired by Bayesian Model (Bishop 2006). The prior probability is similar to the prior probability of Bayesian Model. Meanwhile, as shown in Fig. 2 (a), the input of BACN is followed with two convolution operation to learn a temporary weight vector, which is similar to the likelihood probability of Bayesian Model. The prior probability and the likelihood probability are fused through a feature concatenate operation to generate a feature which encode the information of posterior probability. After that, the posterior feature is followed by two convolution and a softmax operations to produce weight vector. By incorporating prior information, our network is easier to obtain better classification results.

Since the proposed Guided Loss can flexibly control the bias on precision and recall by using different Fn-measure as guidance, we can naturally build a cascade network by Guided Loss to progressively refine the performance. Specifically, as shown in Fig. 2 (b), we first use a 12-layer hybrid attention blocks as feature extraction module to extract the feature for each correspondence. Then a coarse result (coarse result1 in Fig. 2 (b)) can be obtained through these features by $\rm{Fn_{1}}$-measure Guided Loss. Then two refinement modules are followed to perform local optimization to refine the coarse result. Each refinement module is made up of a 3-layer HA Block. Different from feature extraction module, the global context in refinement module is extracted from the coarse result of the previous module instead of all of the correspondences. Besides, in order to gradually optimize the coarse result, the loss function will also progressively bias the precision. The coarse result2 is obtained by $\rm{Fn_{2}}$-measure Guided Loss, and the final result is obtained by $\rm{Fn_{3}}$-measure Guided Loss. During training, $n_{1}>n_{2}>n_{3}$ holds so that the network gradually obtains result with higher precision. Finally, the $E$ matrix is computed by performing weighted eight-point or RANSAC algorithm on the final result, and it is supervised by a $L2$-$loss$.

Parameter Settings. The network is trained by Adam optimizer (Kingma and Ba 2015) with a learning rate being $10^{-3}$ and batch size being 16. The iteration times are set to 200k. In Eq. 18, the loss weight $\eta_{3}$ is 0 during the first 20k iteration and then 0.1 later. $\eta_{1}$ and $\eta_{2}$ are set to 0.1 during the whole training. The $n_{1}$, $n_{2}$ and $n_{3}$ in Fig. 2 are set to 0.3, 0.25 and 0.2 during training, which leads to best relative pose estimation results.

We compare our network with other state-of-the-art methods, including CN-Net (Moo Yi et al. 2018), PointNet (Qi et al. 2017a), ACN-Net (Sun et al. 2020), NM-Net (Zhao et al. 2019a), $\rm N^{3}$-Net (Plötz and Roth 2018) and OA-Net (Zhang et al. 2019) on both YFCC100M$\&$SUN3D and COLMAP datasets. All the networks are trained with the same setting. Tab. 4 summarizes the correspondence classification and relative pose estimation results. Our method shows improvement of 15.59% and 6.94% over CN-Net (baseline of our network) in terms of F1-measure on both YFCC100M$\&$SUN3D and COLMAP datasets. In terms of pose estimation results, the mAP of our network is also better than CN-Net by more than 10%. Besides, when compared with another network, our method performs best, especially on correspondence classification task, by nearly 5 to 10 % over the current best approach in terms of F1-measure. These experiments demonstrate that the proposed network behaves favorably to the state-of-the-art approaches.

HA Block. To demonstrate the performance of HA block, we replace the CN Block in the baseline CN-Net (Moo Yi et al. 2018) with the HA block. Both the Bayesian attentive context normalization (BACN) and channel-wise attention (CA) are tested specifically as Tab. 2. As a comparison, we also replace CN Block with ACN Block (Sun et al. 2020) to train the network. Both of the BACN and CA achieve a better result than ACN, and HA block (BACN + CA) achieves an improvement of about 2% over ACN on both correspondence classification and relative pose estimation results.

In order to further verify the performance of the proposed Guided Loss, we record the training curves of the weight, precision and recall in Fig. 3. As shown in Fig. 3 (a), $\lambda$ in the Guided Loss is dynamically changed, while $\lambda$ in IB-CE-Loss is set to fixed value 0.5. As a result, the Guided Loss can achieve a balance between precision and recall, as shown in Fig. 3 (b). Meanwhile, when using F1-measure, which considers precision and recall equally, as the guidance, the gap between precision and recall is always small. And when using F2-measure, which is more bias towards recall, the recall is always higher than precision. It shows that the result of Guided Loss always accords with the guided Fn-measure, which verifies the effect of the guidance.

Meanwhile, we train the CN-Net (Moo Yi et al. 2018) with the different loss functions (Deng et al. 2018; Zhao et al. 2019b; Lin et al. 2017) and precision, recall, F1-measure and mAP under $5^{\circ}$, $10^{\circ}$ are reported in Tab. 3. As discussed in Introduction, when using Fn-measure as objective function, some relaxation has to be made and not all of the samples are utilized for back propagation. Therefore, Fn-Loss does not even perform as well as IB-CE-Loss. For the proposed Guided Loss, the network can achieve a better result than the other loss functions. This is because the Guided Loss can maintain the advantages of IB-CE-Loss while achieving a balance between precision and recall.

Theorem. Given two equations as follows:

where $\partial l_{X}$ and $\partial l_{Y}$ are the partial derivatives of loss ($loss$) with respect to $X$ and $Y$, and $\partial F_{X}$ and $\partial F_{Y}$ are the partial derivatives of Fn-measure ($Fn$) with respect to $X$ and $Y$. $dloss$ and $dFn$ are the derivatives of $loss$ and $Fn$.

Then, Eq. 20 is a sufficient condition of Eq. 21 .

Proof. As introduced in the paper, Fn-measure ($Fn$) and loss ($loss$) are both the dependent variables of X and Y. Specifically, the form of loss is as follows:

where $N_{pos}$ and $N_{neg}$ are the number of positive and negative samples. We compute the average loss terms of $TP$, $TN$, $FP$ and $FN$ samples respectively, denoted as $l_{TP},l_{TN},l_{FP},l_{FN}$. Then the $loss$ can be transformed as follows:

Since both the $TP$ and $FN$ samples belong to positive class (ground truth is positive), the loss term of each sample are computed by $-log(y_{i})$ in Eq. 22 , where $y_{i}$ ($0\leq y_{i}\leq 1$) is the logit value. In fact, if the logit value of a positive sample (ground truth is positive) is greater than 0.5, then it is a $TP$ sample. And if the logit value is smaller than 0.5, it is a $FN$ sample. Obviously, since $-log(y_{i})$ is a monotone decreasing function, then the loss term of each $FN$ sample is greater than $TP$ sample. Thus, the average loss of $FN$ samples is greater than that of $TP$ samples, i.e.,

Similarly, the the average loss of $FP$ samples is greater than that of $TN$ samples, i. e.,

Then, we can compute the partial derivatives of loss with respect to $X$ and $Y$ from Eq. 23, as follows:

According to the constraints of Eq. 24 and 25, we can obtain the following constraints:

We then perform the same operations on $Fn$ to obtain the constraints of $Fn$. Specifically, $Fn$ is related with precision ($P$) and recall ($R$), and $P$, $R$ are both related with $X$, $Y$, as follows:

According to the compound derivation formula, we can compute the $\partial F_{X}$ and $\partial F_{Y}$ as follows:

The $\partial F_{P}$, $\partial F_{R}$, $\partial P_{X}$, $\partial P_{Y}$ all can be computed from Eq. 28 and 29, as follows:

We can easily obtain the following constraints of $\partial F_{X}$ and $\partial F_{Y}$ from Eq. 30, 31:

Meanwhile, since Fn-measure ($Fn$) and loss ($loss$) are both the dependent variables of X and Y, the differential form of $loss$ and $Fn$ in Eq. 21 can be expressed as follows:

Thus,

According to the constraints of Eq. 27 and 32, we can further expand Eq. 34 as follows:

If Eq. 20 holds, then,

then,

Thus, we can proof that Eq. 20 is a sufficient condition of Eq. 21.

Guided Loss with other baseline networks. We further analyze our Guided Loss by replacing the classification loss functions of other models with Guided Loss. We first train three recent networks, including CN-Net (Moo Yi et al. 2018), ACN-Net (Sun et al. 2019) and NM-Net (Zhao et al. 2019a), with their original classification loss. Then we replace their classification loss with our Guided Loss. The results are reported in Tab. 4. Each network with the supervision of our loss can lead to a result with more balanced precision and recall and a higher F1-measure. As a result, the each network increase the mAP by 1-3% without modifying anything.

Parameter $n$ of Guided Loss. Our Guided Loss is designed to build a perfect negative correlation relationship between cross entropy loss and Fn-measure for better optimizing Fn-measure. To verify that our guidance actually works, we train the CN-Net (Moo Yi et al. 2018) with different Fn-measure Guided Loss, respectively. Meanwhile, we also train the CN-Net with the cross entropy loss (CE-Loss) and Instance Balance cross entropy loss (Deng et al. 2018; Moo Yi et al. 2018) (IB-CE-Loss). The classification result are shown in Tab. 5. We record the precision (P), recall (R), and five Fn-measures (n = 0.5, 1, 1.5, 2, 2.5 respectively) of the classification results on test dataset. As shown in Tab. 5, with the guidance of a specific Fn-measure, our Guided Loss can achieve the best performance on this measurement. It shows the direct guidance of our loss. We can choose the corresponding Fn-measure to guide the loss according to the requirements of precision and recall of different tasks.

Visual Result. In order to show the classification result of our network, we visualize the matching results of RANSAC (Fischler and Bolles 1981), CN-Net (Moo Yi et al. 2018) and our network in Fig. 4.