VisIRNet: Deep Image Alignment for UAV-taken Visible and Infrared Image Pairs

Recent advancements in Unmanned Aerial Vehicle (UAV) technologies, computing and sensor technologies, allowed use of UAVs for various earth observation applications. Many UAV systems are equipped with multiple cameras today, as cameras provide reasonable and relatively reliable information about the surrounding scene in the form of multiple images or image pairs. Such image pairs can be taken by different cameras, at different view-points, different modalities or at different resolutions. In such situations, the same objects or the same features might appear at different coordinates on each image and, therefore, an image alignment (registration) step is needed prior to applying many other image based computer vision applications such as image fusion, object detection, segmentation or object tracking as in [45, 47, 46].

The infrared spectrum and visible spectrum may reflect different properties of the same scene. Consequently, images taken in those modalities, typically, differ from each other. On many digital cameras, the visible spectrum is captured and stored in the form of Red-Green-Blue (RGB) image model and a typical visible spectrum camera captures visible light ranging from approximately 400 nanometers to 700 nanometers in wavelength [7, 8, 37]. Infrared cameras, on the other hand, capture wavelengths longer than those of visible light, falling between 700 nanometers and 10000 nanometers [11]. Infrared images can be further categorized into different wavelength ranges as in near-infrared (NIR), mid-infrared (MIR), and far-infrared (FIR) capturing different types of information in the spectrum [11, 12, 17, 18].

Image alignment is, essentially, the process of mapping the pixel coordinates from different coordinate system(s) into one common coordinate system. This problem is studied under different names including image registration and image alignment. We will also use the terms alignment and registration interchangeably in this paper. Typically, alignment is done in the form of image pairs mapping from one image (source) onto the other one (target) [20]. Image alignment is a common problem that exists in many image-based applications where both of the target and source images can be acquired by sensors using the same modality or using different modalities. There is a wide range of applications of image alignment in many fields including medical imaging [1, 23], UAV applications [38, 24], image stitching  [10] and remote sensing applications [5, 32, 31, 6, 39].

Image alignment, in many cases, can be reduced to the problem of estimating the parameters of the perspective transformation between two images acquired by two separate cameras, where we assume that the cameras are located on the same UAV system. Fig. 1 summarizes such an image alignment process where the input consists of a higher resolution RGB image (e.g., $192\times 192$ pixels) and a lower resolution IR image (e.g., $128\times 128$ pixels visualized in pseudocolors in the figure). The output of the registration algorithm is the registered (aligned) IR image on the RGB image’s coordinate system. As perspective transformation [22] is typically enough for UAV setups containing nearby onboard cameras, our registration process uses a registration function based on the Homography (H) matrix. H contains 8 unknown (projection) parameters and the goal of the registration process is predicting those 8 unknown parameters, directly or indirectly.

In the relevant literature, registering RGB and IR image pairs is done by using both classical techniques (such as Scale-Invariant Feature Transform, SIFT, [35] along with the Random Sample Consensus, RANSAC, [16] algorithm as in [3]) and by using more recent deep learning based techniques as in [36, 9, 54]. Classical techniques include feature-based [42, 52] and intensity-based  [41] methods. Feature-based [42, 52] methods essentially find correspondences between the detected salient features from images [49]. Salient features are computed by using approaches such as SIFT  [34], Speeded-Up Robust Features (SURF) [4], Harris Corner [21], Shi-Tomas corner detectors [26] in each image. The features from both images are then matched to find the correspondences as in [48, 44, 43], and to compute the transformation parameters in the form of homography matrix. The RANSAC [48] algorithm is commonly used to compute the homography matrix that minimizes the total number of outliers in the literature. Intensity-based  [41] methods compare intensity patterns in images via similarity metrics. By estimating the movement of each pixel, optical flow is computed and used to represent the overall motion parameters. In  [15, 2] uses LK based algorithms that take the initial parameters and iteratively estimate a small change in the parameters to minimize the error. A typical intensity-based registration technique, essentially uses a form of similarity as its metric or as its registration criteria including Mean Squared Error (MSE) [19], cross-correlation [30], Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR) [53]. Such metrics are not sufficient when source image and target image are acquired by different modalities. This can yield poor performance when such intensity based method are used.

Overall, such major classical approaches, typically, are based on finding and matching similar salient keypoints in image pairs, and therefore, they can yield unsatisfactory results in various multi-modal registration applications.

Relevant deep alignment approaches are using a form of keypoint matching, template matching or Lukas-Kanade (LK) based approaches as in [29, 9]. Those techniques typically consider multiple points or important regions in images to compute the homography matrix H which contains the transformation parameters. However, having the information of four matching points represented by their corresponding 2D coordinates $(x_{i},y_{i})$, where $i=1,2,3,4$ is sufficient to estimate H. Therefore, if found accurately, four matching image-corner points between the IR and RGB images would be enough to perform accurate registration between the IR and RGB images. While many techniques based on keypoint extraction can be employed to find matching keypoints between the images, we argue that the corner points on the borders of one image can also be considered as keypoints, and by using those corners of the image, we do not need to utilize any keypoint extraction step.

In this paper, we propose a novel deep approach for registering IR and RGB image pairs, where instead of predicting the homography matrix directly, we predict the location of the four corner points of the entire image directly. This approach removes the additional iterative steps introduced by LK based algorithms and eliminates the steps of computing and finding important keypoints. Our main contributions can be listed as follows: (i) we introduce a novel deep approach for alignment problems of IR images onto RGB images taken by UAVs, where the resolutions of the input images differ from each other; (ii) we introduce a novel two-branch based deep solution for registration without relying on the Lukas-Kanade based iterative methods; (iii) instead of predicting the homography matrix directly, we predict the corresponding coordinates of the four corner points of the smaller image on the larger image; (iv) we study and report the performance of our approach on multiple aerial datasets and present the state of the art results.

Many recent techniques performing image alignment rely on deep learning. Convolutional Neural Networks (CNNs) form a pipeline of convolutional layers where filters learn unique features at distinct levels of the network. For example, the authors of [14] proposed a Deep Image Homography Estimation Network (DHN) that uses CNNs to learn meaningful features in both images and it directly predicts the eight affine transformation parameters of the homography matrix. Later, the authors of [28] proposed using a series of networks to regress the homograph parameters in their approach. The latter networks in their proposed architecture aims to gradually improve the performance of the earlier networks. Their method builds on top of DHN [14]. Another work in [9], proposed incorporating the LK algorithm in the deep learning pipeline.

The authors of [54] used a CNN based network and introduced a learning based Lucas-Kanade block. In their work, they designed modality specific pipelines for both source and template images, respectively. At the end of each block, there is a unique feature construction function. Instead of using direct output feature maps, they constructed features based on Eigen values and Eigen vectors of the output feature maps. The features constructed from the source and template network channels have a similar learned representation. Transformation parameters found at a lower scale are given as in input to the next level and the LK algorithm iterates until a certain threshold is reached. In another work, the authors of  [13] utilized disentangled convolutional sparse coding to separate domain-specific and shared features of multi-modal images for improved accuracy of registration. Multi-scale Generative Adversarial Networks (GANs) are also used to estimate homography parameters as in  [36].

The architectural comparisons of the above-mentioned multiple networks is provided in Fig.2. In DHN [14], the image to be transformed (it is noted as I_IR in the figure) is padded to have the same dimensions as the target image (I_RGB) and they are concatenated channel-wise. The concatenated images are given to the Deep Homography Network (DHN) for the direct regression of the 8 values of the homography matrix. On the other hand, Multi-scale homography estimation (MHN) [28] adapts using a series of networks (Net_i). The inputs for Net₂ are a concatenation of I_IR and I_RGB. For the succeeding levels, first, the warping function performs the projective inverse warping operation on the infrared image (I_IR) via the homography matrix which was predicted at the previous level. The resulting image (I${}^{{}^{\prime}}_{IR_{i}}$) is first concatenated with I_RGB and then given as input to the Net_i. For the following levels, the current matrix and previously predicted matrices are multiplied to form the final prediction. This way MHN aims to learn correcting mistakes made in the earlier levels. Cascaded Lucas-Kanade Network (CLKN) [9] uses separate networks for each modality. They use levels of different scales in the form of feature pyramid networks, and perform registration from smallest to the largest. The homography matrix from the earlier LK-layer is given as input to the next. Deep Lucas-Kanade Feature Maps (DLKFM) [54] also performs coarse to fine registration as shown in Fig.2. It uses a special feature construction block called (fcb). The (fcb) block takes in the feature maps and transforms them into new feature based on the Eigen vectors and covariance matrix. The constructed features capture principal information and the registration is performed on the constructed feature maps, thus, it aims to increase the accuracy of the LK-layer. Our approach uses separate feature embedding blocks to process each modality separately. It is trained to extract modality specific features so that the output feature maps of different modalities can have similar feature representations.

In our proposed approach, we aim at performing accurate, single and multi-modal image registration which is free of the iterative nature of LK-based algorithms. We name our network VisIRNet, where we aim to predict the location of the corners of the input image on the target image directly, since having four matching points is sufficient to compute the homography parameters. In our proposed architecture, we assume that there are two input images with different resolutions. The overview of our architecture is given in Fig.3. Our approach first processes two inputs separately by passing them through their respective feature embedding blocks and extracts representative features. Those features are then combined and given to the regression block as input. The goal of the regression block is computing the transformation parameters accurately. The output of the regression block is eight dimensional (which can represent the total number of homography parameters or the coordinates of the four corner points of the source image on the target image).

In this section, we describe our experimental procedures, used datasets and our metrics. Below we describe our used datasets.

For each dataset that we use, we generate the training and test sets as follows: (i) Select a registered image pair at higher resolutions. (ii) Sample (crop) regions around the center of the image to get smaller patches of 192x192 pixels. This process is done in parallel for visible and infrared images. (iii) If the extracted patches are not sufficiently aligned, manually align them. (iv) For each pair, select a subset of the IR image, by randomly selecting 4 distinct locations on the image. (v) Find perspective transformation parameters that map those randomly chosen points to the following fixed locations: (0,0) , ($n-1$,0) , ($n-1,n-1$) , (0,$n-1$) so that they can correspond to the corners of the unregistered IR image patch; where we assume that the unregistered $I_{IR}$ is $n\times n$ dimensional (in our experiments $n$ is set to 128). This process creates an unregistered infrared patch (from the already registered ground truth) that needs to be placed back to its true position. (vi) Use those 4 initially selected points as the ground truth corners for the registered image. (vii) Repeat process $k$ times to create $k$ different image pairs. This newly-created dataset is then split into training and test sets. (viii) The RGB images are used as the target set and the transformed infrared patches are used as the source set (for both training and testing). This process is done on randomly selected registered pairs for each dataset. Fig.4 also illustrates this process on a pair of RGB and IR images. The list of all the used data sets and their details are summarized in Table IV.

Table III shows an ablation study on using different loss functions in each block in our architecture. The used metric in the table is Ace and the best values are shown in bold. The loss functions in each row are used to train the MMFEB block includes $\mathcal{L}_{sim}$, $\mathcal{L}_{MAE}$ and $\mathcal{L}_{SSIM}$. The loss functions used for the regression block are $\mathcal{L}_{Ace}$ and ${L}^{H}_{2}$. In the table, the last column shows the average error for both $\mathcal{L}_{Ace}$ and ${L}^{H}_{2}$ (over two datasets including SkyData and VEDAI) for each of the used loss functions in the MMFEB block.

Next, we provide experimental results on the effect of the hyperparameters that we studied for both ModelA and ModelB. Table V summarizes those results. In particular, we studied the effect of using different loss functions ($L_{1}$, $L_{2}$ and $L_{Ace}$) and using different batch sizes for both models. All the experiments were done on the SkyData set. The best results are shown in bold. Overall, ModelB showed promising results achieving better results when compared to the ModelA. Therefore, for the rest of our experiments, we kept using ModelB only.

Fig.6 uses a box plot, also known as a box-and-whisker plot, to display the distribution of average corner error for different different datasets and for different models. It provides a summary of key statistical measures such as the minimum, first quartile (Q1), median (Q2), third quartile (Q3), and maximum. The length of the box indicates the spread of the middle 50% of the data. The line inside the box represents the median (Q2). The whiskers extend from the box and represent the variability of the data beyond the quartiles, in our case they represent $Q1-1.5*IQR$ and $Q3+1.5*IQR$. Individual data points that lie outside the whiskers are considered outliers and are plotted with diamonds. The figure compares the results for 6 algorithms on 5 different datasets.

Fig.7 shows the performance of 6 methods (SIFT, DHN, MHN, CLKN, DLKFM, Ours) on the SkyDataV1 dataset, in terms of average corner error. Skydata has RGB and infrared image pairs. In this figure, we aim to show that feature based registration techniques such as SIFT perform poorly whereas methods that leverage neural networks and learn representations are superior.

Fig.5 gives detailed qualitative results of our experiments. Each row represent a sample taken from a different dataset. The columns represent inputs and results for different approaches. Target is (192x192) (first column), and source (second column) (128x128) are input image pairs. Warped (third column), is the ground truth projection of source to the coordinate system of target image and Registered (fourth column) is the warped image overlayed on the target image as shown. Columns from 5 to 10 shows the registered and overlayed results for SIFT, DHN, MHN, CLKN, DLKFM, and Ours (ModelA) for the given input pair. While almost all algorithms relatively done well on Google Earth pair (which provides similar modalities for both target and source images), when the modalities are significantly different, as in the SkyData, Google Maps and VEDAI pairs, the figure shows that SIFT, CLKN, MHN, DHN and DLKFM algorithms can struggle for aligning them and they may not converge to any useful result near the ground truth (see SIFT and CLKN results), while our approach converges to the ground truth by yielding small ACE error for each of those sample pairs.

Table VI illustrates the results of using different approaches for each dataset, separately. in Table VI(e), the MSCOCO results being a single modality dataset, SIFT perform relatively better but there are cases where the algorithm could not find homography due to insufficient pairs. Google earth in (c) also has RGB image pairs but from different seasons. SIFT algorithm is still able to pick enough salient features therefore the performance is still reasonable. (d) Google maps, (a) SkyData and (b) VEDAI have pairs of significant modality difference. Deep learning based approaches were able to perform registration often with high number of outliers. Our approach was able to perform registration on both single and multi-modal image pairs, specifically we were able to keep the max error minimum as opposed to LK-based approaches.

In this paper, we introduce a novel image alignment algorithm that we call VisIRNet. VisIRNet has two branches and does not have any stage to compute keypoints. Our experimental results show that our proposed algorithm performs state of the art results, when it is compared to the LK based deep approaches.

Our method’s main advantages can be listed as follows: (a) Number of iterations during inference: Above-mentioned Lucas-Kanade based methods (after the training stage), also iterate a number of times during the inference stage and at each iteration, they try minimizing the loss. However, those methods are not guaranteed to converge to the optimal solution and often number of iterations, chosen as a hyperparameter, is an arbitrary number during the inference stage. Such iterative approaches introduce uncertainty for the processing time, as convergence can happen after the first iteration in some situations and after the last iteration in other situations during inference. Such uncertainty also affects the real time processing of images, as they can introduce varying frame per second values. Our Method uses a single pass during inference with make it more applicable to real time applications.

(b) Dependence on the initial H estimate: In addition to the above-mentioned difference, the LK-based algorithms require an initial estimate of the homography matrix and the performance (and number of iterations required for convergence) directly depend on the initial estimate of H and therefore it is typically given as input (hyperparameter). While we also have initialization of the weights in our architecture, we do not need an initial estimate of the homography matrix within the architecture as input.

Image alignment on image pairs taken by different onboard cameras on UAVs is a challenging and important topic for various applications. When the images to be aligned acquired by different modalities, the classic approaches, such as SIFT and RANSAC combination, can yield insufficient results. Deep learning techniques can be more reliable in such situations as our results demonstrate. LK based deep techniques recently shown promise, however, we demonstrate with our approach (VisIRNet) that without designing any LK based block, and by focusing only on the four corner points, we can sufficiently train deep architectures for image alignment.

This paper has been produced benefiting from the 2232 International Fellowship for Outstanding Researchers Program of TÜBİTAK (Project No:118C356). However, the entire responsibility of the paper belongs to the owner of the paper.

Sedat Ozer received his M.Sc. degree from Univ. of Massachusetts, Dartmouth and his Ph.D. degree from Rutgers University, NJ. He has worked as a research associate in various institutions including Univ. of Virginia and Massachusetts Institute of Technology. His research interests include pattern analysis, remote sensing, object detection & segmentation, object tracking, visual data analysis, geometric and explainable AI algorithms and explainable fusion algorithms. As a recipient of TUBITAK’s international outstanding research fellow and as an Assistant Professor, he is currently at the department of Computer Science at Ozyegin University.

Alain Patrick Ndigande received his B.Eng. degree from Kocaeli University, Turkey, in 2022. He is currently a M.Sc. student at Ozyegin University. His current research interests are deep learning, image registration and remote sensing.