An invariant feature extraction for multi-modal images matching.

Harris corner detection [11] is an effective keypoints extraction method with stability and robustness, which has been widely used in multi-modal image matching. However, for multi-source images, the textures are likely to be very different, such as optical image and digital map as shown in Fig.2(a). There are texture features in the optical image that do not exist in the digital map. Due to the influence of inconsistent texture information, the keypoints are difficult to locate in the same position, causing the repetition rate of points very low, as shown in Fig.2(b). It is hoped that the extracted feature points are more focused on the salient and stable structure of the image. Thus the idea of adopting image’s phase congruency is come up with.

Phase congruency (PC) has been proved to be robust under different imaging modalities [7, 8, 9], which extracts stable structure features. The 2D-PC model can be calculated using components at multiple scales $s$ and orientations $o$ of the LogGabor wavelet [12, 13]:

$${\bf{PC}}(x,y)=\frac{{\sum\nolimits_{s}{\sum\nolimits_{o}{{\omega_{o}}(x,y)\lfloor{{\bf{A}}_{s,o}}(x,y)\Delta{{\bf{\phi}}_{s,o}}(x,y)-T}\rfloor}}}{{\sum\limits_{s}{\sum\limits_{o}{{{\bf{A}}_{s,o}}(x,y)}}+\varepsilon}}$$

(1)

where ${\omega_{o}}(x,y)$ is the weighting factor based on frequency spread; ${\bf{A}}_{s,o}$ is amplitude component of LogGabor response; $\Delta{{\bf{\phi}}_{s,o}}$ is the phase deviation; T is the noise compensation; $\lfloor\cdot\rfloor$ is a truncation function that produces the equal when positive and zero otherwise. A maximum moment map ${{\bf{M}}_{\psi}}$ which indicates edge features and a minimum moment map ${{\bf{m}}_{\psi}}$ which indicates corner features are then calculated as:

$${\bf{a}}=\sum\nolimits_{o}{{{({\bf{P}}{{\bf{C}}_{{\theta_{0}}}}\cos({\theta_{0}}))}^{2}}}$$

(2)

$${\bf{b}}=2\sum\nolimits_{o}{({\bf{P}}{{\bf{C}}_{{\theta_{0}}}}\cos({\theta_{0}}))\cdot({\bf{P}}{{\bf{C}}_{{\theta_{0}}}}\sin({\theta_{0}}))}$$

(3)

$${\bf{c}}={\sum\nolimits_{o}{({\bf{P}}{{\bf{C}}_{{\theta_{0}}}}\sin({\theta_{0}}))}^{2}}$$

(4)

$${\bf{\psi}}=\frac{1}{2}\arctan(\frac{{\bf{b}}}{{{\bf{a-c}}}})$$

(5)

$${{\bf{M}}_{\psi}}=\frac{1}{2}({\bf{c}}+{\bf{a}}+\sqrt{{{\bf{b}}^{2}}+{{({\bf{a}}-{\bf{c}})}^{2}}})$$

(6)

$${{\bf{m}}_{\psi}}=\frac{1}{2}({\bf{c}}+{\bf{a}}-\sqrt{{{\bf{b}}^{2}}+{{({\bf{a}}-{\bf{c}})}^{2}}})$$

(7)

where ${\bf{P}}{{\bf{C}}_{{\theta_{0}}}}$ is the PC map at orientation $\theta_{0}$. Then the two feature maps are added together to get a single map containing both edge and corner features:

$${{\bf{M}}_{w}}={{\bf{M}}_{\psi}}+{{\bf{m}}_{\psi}}$$

(8)

Shi-Tomasi [14] feature is an improved algorithm of Harris, which states that the stability of the corner is related to the smaller eigenvalue of the feature matrix in Harris. Therefore, the Shi-Tomasi detector is performed on the feature map to extract stable feature points:

$$cornerness=min(\lambda_{1},\lambda_{2})$$

(9)

$$\textbf{\emph{M}}=\left[{\begin{array}[]{*{20}{l}}{\sum\limits_{\bf{W}}{{{{\bf{M}}_{w}}_{x}}^{2}}}&{\sum\limits_{\bf{W}}{{{{\bf{M}}_{w}}_{x}}{{{\bf{M}}_{w}}_{y}}}}\\ {\sum\limits_{\bf{W}}{{{{\bf{M}}_{w}}_{x}}{{{\bf{M}}_{w}}_{y}}}}&{\sum\limits_{\bf{W}}{{{{\bf{M}}_{w}}_{y}}^{2}}}\end{array}}\right]$$

(10)

where $\lambda_{1},\lambda_{2}$ are the eigenvalues of $\bf{M}$, ${{{\bf{M}}_{w}}_{x}}$ and ${{{\bf{M}}_{w}}_{y}}$ are the gradient of the weighted moment map ${{\bf{M}}_{w}}$ in the $x$ and $y$ directions, and ${\bf{W}}$ is a gaussian window. After filtering the images using Eq.(1)-(10), with local non-maximum suppression and threshold judgment, the keypoints are obtained from the multi-modal images, as shown in Fig.2(c), the repetition rate of which is highly improved.

So far, there have been many studies on multi-modal image matching techniques, and a variety of robust features have been proposed, all of which have their effects and advantages. The key problem lies in finding the invariant or similar features among images with different modals. We reinspect the characteristics and matching methods of multi-modal image data and condense a core idea that is analyzed below to guide our feature design.

To intuitively show the characteristics of the multi-modal robust feature, we briefly summarize the common intensity distortion in multi-modal images, as shown in Fig.3. The four images can be taken as the edge of an object or the interface of two substances. Assume that the center point is the keypoint, and the image block represents the intensity information of its neighborhood. In Fig.3, the intensity amplitude on the left is lower than on the right. In Fig.3, the gradient amplitude of the two parts changes. In Fig.3, due to large intensity distortion, the gradient orientation is reversed. And Fig.3 represents a general situation, which is considered as non-linear intensity distortion or some degradation such as down-sampling or blurring. Notice that the gradient orientation remains on the same line, and for the reversion case, if the angle is limited to [-90°, 90°), the orientation is still 0°.

In multi-source images, due to differences in sensors, temporal, environments, etc., various intensity distortions may be caused, but the morphology of the most detailed part of the image basically lies in the four cases in Fig.3. In summary, the magnitude of the local gradient varies, but the orientation is basically stable, which defines the feature information that should be focused on. In MS-HLMO [10], a similar idea emerges, which prompts it to count only the gradient orientation in its PMOM feature map and abandon the amplitude. However, PMOM is still calculated using classic gradient that is very sensitive to image noise and inconsistency. So, in this research, the orientation is obtained with an odd-LogGabor filter. The odd-LogGabor wavelet has been widely used as a representation of gradient operator in image processing, which has much stronger robustness. The image’s gradients along x and y directions based on odd-LogGabor are then calculated as:

$$\left\{{\begin{array}[]{*{20}{c}}{{\bf{G}}_{x}^{{\rm{LG}}}(x,y)=\sum\nolimits_{s}{\sum\nolimits_{o}{{\bf{I}}(x,y)*{\bf{LG}}_{s,o}^{{\rm{odd}}}(x,y)\cdot\cos(o)}}}\\ {{\bf{G}}_{y}^{{\rm{LG}}}(x,y)=\sum\nolimits_{s}{\sum\nolimits_{o}{{\bf{I}}(x,y)*{\bf{LG}}_{s,o}^{{\rm{odd}}}(x,y)\cdot\sin(o)}}}\end{array}}\right.$$

(11)

where ${\bf{LG}}_{s,o}$ denotes an odd-LogGabor filter with scale $s$ and orientation $o$. The odd-LogGabor responses ${\bf{G}}_{x}^{{\rm{LG}}}(x,y)$ and ${\bf{G}}_{y}^{{\rm{LG}}}(x,y)$ is taken as the representation of gradients and in average squared gradient (ASG) calculation:

$$\left[\begin{array}[]{l}{{\bf{G}}_{{W_{\sigma}},s,x}}\\ {{\bf{G}}_{{W_{\sigma}},s,y}}\end{array}\right]=\left[\begin{array}[]{l}\sum\limits_{{W_{\sigma}}}{{{({\bf{G}}_{x}^{{\rm{LG}}})}^{2}}-{{({\bf{G}}_{y}^{{\rm{LG}}})}^{2}}}\\ \sum\limits_{{W_{\sigma}}}{2{\bf{G}}_{x}^{{\rm{LG}}}{\bf{G}}_{y}^{{\rm{LG}}}}\end{array}\right]$$

(12)

Another issue is that, according to visual saliency, the pixel closer to the center pixel has more importance. So an improvement is made to the PMOM that in a local area, features from a larger scale should have fewer effects. Each scale is given a weight that is inversely proportional to the scale, then the local orientation is calculated as:

$${\bf{G}}_{{\rm{PMOM}}}^{{\rm{LG}}}=\frac{1}{2}\angle(\sum\limits_{\sigma}{\frac{1}{\sigma}{{\bf{G}}_{{W_{\sigma}},s,x}}},\sum\limits_{\sigma}{\frac{1}{\sigma}{{\bf{G}}_{{W_{\sigma}},s,y}}})$$

(13)

$$\angle(X,Y)=\left\{\begin{array}[]{l}\arctan(\frac{Y}{X}),X\geq 0\\ \arctan(\frac{Y}{X})+\pi,X<0,Y\geq 0\\ \arctan(\frac{Y}{X})-\pi,X<0,Y<0\end{array}\right.$$

(14)

This orientation feature map ${\bf{G}}_{{\rm{PMOM}}}^{{\rm{LG}}}$ based on odd-LogGabor and weighted-PMOM (WPMOM) has much stronger invariance and stability in multi-modal images. The schematic diagram of feature extraction process is shown in Fig4.

After the most crucial invariant feature map is obtained, the next step is to extract a descriptor for each keypoints with a selected structure. Theoretically, any kind of valid descriptor structure is feasible. In the proposed framework, the generalized gradient location and orientation histogram-like (GGLOH) feature descriptor [10] is adopted for feature description, which has proved a better performance and is more operable. The WPMOM value at each keypoint is taken as the reference orientation, and the WPMOM values within the local area are counted with GGLOH to obtain descriptor vectors of the keypoints which are then for matching.

To deal with scale differences, the feature extraction is extracted in the scale-space by using Gaussian pyramid. The multi-scale strategy proposed in [10] has shown effective results, which is adapted to the registration process. The commonly used nearest neighbor (NN) matching and fast sample consensus (FSC) outlier removal are utilized for the proposed algorithm and also for each method in the experiments to get a fair comparison.