End-to-End Learning for Simultaneously Generating Decision Map and Multi-Focus Image Fusion Result

Handcrafted feature based fusion algorithms concentrate on the profound image analysis of transform or spatial domains. Transform domain based algorithms adopt decomposed coefficients from a selected transform domain to measure different activity levels in the input source images, such as laplacian pyramid (LP) (Burt and Adelson 1983) and non-subsampled contourlet transform (NSCT) (Zhang and long Guo 2009). Spatial domain based algorithms measure activity levels with gradient features, such as spatial frequency (Li, Kwok, and Wang 2001), multi-scale weighted gradient (MWG) (Zhou, Li, and Wang 2014), and dense SIFT (DSIFT) (Liu, Liu, and Wang 2015).

Deep learning based algorithms provide prevalent solutions to image fusion problems. CNN-Fuse (Liu et al. 2017) first used a convolutional network to automatically learn features in each patch of image and decided which patch is the clarity region, which achieved better performance compared to handcrafted feature based algorithms. Afterward, some researchers tried to modify the network to improve the fusion quality or efficiency. Han Tang proposed a pixel-wise fusion CNN to further improve the fusion quality (Tang et al. 2018). Dense-Fuse (Li and Wu 2019), U2Fusion (Xu et al. 2020b), and SESF-Fuse (Ma et al. 2019) fused images in the unsupervised training procedure. IFCNN (Zhang et al. 2020b) presented a general image fusion framework to handle different kinds of image fusion tasks. However, there are still other parts of deep learning based algorithm that need to refine.

The output mode is an important module in network designing. Some algorithms tried to use a decoder to directly output the fused result. Hao Zhang (Zhang et al. 2020a) used only one convolutional layer in decoder to fuse multi-scale features and generate fused result. To improve the reconstructive ability, Hyungjoo Jung (H et al. 2020) used residual block to improve the efficiency of gradient propagation, and some works (Huang et al. 2020; Zhang et al. 2021) used generative adversarial network to automatically ameliorate fusion quality. However, due to nonlinear mapping in the decoder, these structures cannot precisely reconstruct fused result. This leads to relatively unrealistic performance in fusion evaluation. Therefore, some structures resorted to generate an intermediate DM, to decide which pixel should appear in fused result. Some works (Liu et al. 2017; Li et al. 2020) used CNN to directly output DM. SESF-Fuse (Ma et al. 2019) used spatial frequency to calculate gradient in deep features and draw out DM. Han Xu  (Xu et al. 2020a) used a binary gradient relation map to further ask decoder to preserve gradient information in DM. Despite the highly fusing quality of these structures, they need some post-processing methods (or consistency verification) with empirical parameters to rectify the DM, such as morphology operations (opening and closing calculation) and small region removal strategy, which limits the generalization of the structure to different scenes of image fusion.

The objective function is a key point in structure optimization. In the field of multi-focus image fusion, the gradient in source images is an important factor to decide which part of the image is clear. However, many deep learning structures only used the l2 norm and SSIM objective function to optimize the network (Li and Wu 2019; Ram Prabhakar, Sai Srikar, and Venkatesh Babu 2017), which did not ask the network to preserve the gradient information in fused image. Hyungjoo Jung (H et al. 2020) proposed structure tensor to preserve the overall contrast of images. Jinxing Li (Li et al. 2020) used an edge-preserving loss function to preserve gradient information, but it only considered gradient intensity and not took orientation information into account. In this work, we try to modify the commonly used classical gradient based evaluation metric as the loss function to directly optimize the network to export clearly fused result.

Most applications of multi-focus fusion are based on multiple images. However, almost multi-focus fusion structures concentrated on two images scene and only used one by one serial fusion strategy for multiple images (Liu et al. 2017; Xu et al. 2020b), which has in-acceptable time consumption. To the best of our knowledge, we are the first work to concentrate on the implementation efficiency in multiple images fusion scene.

The overall fusion network structure is shown in Figure 1. It includes two paths of convolutional operations, feature extraction and decision. First, we use the feature extraction path to collect multi-scale deep features of each source image. Second, we take the spatial frequency (SF) module to calculate activity level of each scale. Third, in the decision path, we concate multi-scale activity levels and feed them into some convolutional operations to draw out the initial DM, which records the probability of each pixel should be in-focused in each source image. Then we apply guided filter (He, Sun, and Tang 2013) to smooth the boundary of DM and acquire final DM. Finally, we cascade the fusion module in our structure and generate the fusion result.

We define a gradient aware loss function to optimize the network to simultaneously output DM and clear fusion result. The final loss function is defined in Eq 5.

where $\lambda$ is a weight to balance the importance between two losses, and $\lambda=1$ in this work.

$L_{Dice}$ is a classical loss function in semantic segmentation (Fausto, Nassir, and Seyedahmad 2016), which is defined in Eq 6.

where the sums run over the $N_{P}$ pixels, of the predicted binary segmentation map $p_{i}\in DM$ and the ground truth map $g_{i}\in G$. Adding 1 is to mitigate the gradient vanishing issue.

In addition, we propose to use $L_{Q_{g}}$ to optimize the network to export the final clear fused result. In the field of multi-focus image fusion, it is commonly speculated that only objects within the DOF have a sharp appearance in a photograph, while others are likely to be blurred (Liu et al. 2017). However, lots of previous works did not consider preserving gradient information in network training. In this work, we focus on a classical gradient based fusion evaluation metric, $Q_{g}$ or $Q^{AB}_{F}$ (Xydeas and Petrovic 2000), and make it differentiable as loss function in an end-to-end training procedure. By using this optimization, we lead the network to preserve gradient information in the final fused result.

$Q_{g}$ is an evaluation metric that measures the amount of edge information transferred from input images to the fused image (Xydeas and Petrovic 2000). Consider two input images $A$ and $B$, and a fused image $F$. A sobel edge operator is applied to yield the edge strength $g_{i}$ and orientation $\alpha_{i}$ of each pixel $i$. Thus, for an input image $A$:

where $s^{Ax}_{i}$ and $s^{Ay}_{i}$ are the respective convoluted results with the horizontal and vertical sobel templates.

The relative strength $G^{AF}_{i}$ and orientation value $\Delta^{AF}_{i}$ between input image A and fused image F are defined as:

Unfortunately, the Heaviside function in Eq 9 and absolute function in Eq 10 are not differentiable and thus cannot be included in training stage. Therefore, we propose to use the sigmoid function as a smooth approximation to the Heaviside function which is defined as:

where $k$ controls the steepness of the curve and closeness to the original Heaviside function, larger $k$ means closer approximation ($k=1000$ in our work). Then, Eq 9 can be rewritten as Eq 12.

And Eq 10 can be rewritten as Eq 13.

Note that in pytorch implementation (Adam et al. 2019), the gradient of absolute function is 0 when input of that equals 0, which is differentiable. Thus it can use Eq 10 rather than Eq 13 in pytorch. The detailed analysis can be found in the experiment section.

The edge strength and orientation preservation values, respectively, can be derived as:

where the constants $\Gamma_{g}$, $k_{g}$, $\sigma_{g}$ and $\Gamma_{\alpha}$, $k_{\alpha}$, $\sigma_{\alpha}$ determine the shapes of the respective sigmoid functions (same with Eq 11) used to form the edge strength and orientation preservation value. Normally, $\Gamma_{g}=\Gamma_{a}=1$, $k_{g}=-10$, $k_{\alpha}=-20$, $\sigma_{g}=0.5$, $\sigma_{\alpha}=0.75$. The edge information preservation value is then defined as:

The final assessment is obtained from the weighted average of the edge information preservation values:

where $w^{A}_{i}=[g^{A}_{i}]^{\gamma}$ and $w^{B}_{i}=[g^{B}_{i}]^{\gamma}$. $\gamma$ is a constant, and usually sets $\gamma$ = 1.

In total, we modify a gradient based classical fusion evaluation metric ($Q_{g}$) as a loss function to optimize the network to export clearly fused result.

We further show an experiment to visualize the comparison of $L_{Dice}+L_{Q_{g}}$ and $L_{Q_{g}}$, as shown in Figure 2. It is shown that the fusion model trained with $L_{Q_{g}}$ have less noise in the decision map compared to the model without it, which means $L_{Q_{g}}$ can act as a post-processing method to improve the fusion quality because it can preserve gradient information in the image.

Most applications of multi-focus fusion are based on multiple images. However, currently almost multi-focus fusion structures concentrated on two images scene and only used one by one serial fusion strategy for multiple images fusion. As shown at the top of Figure 3, one-by-one serial strategy needs to run $2\times(N_{I}-1)$ times feature extraction paths and $N_{I}-1$ times decision paths, where $N_{I}$ is the number of the source images. In this work, we propose a decision calibration strategy, which shown at the bottom of Figure 3. It only needs to run $N_{I}$ times feature extraction paths and $N_{I}-1$ times decision paths by using the calibration module, which can generally decrease time consumption.

In the decision calibration strategy, the first image is used as baseline, and feeds it to the structure with other images. Thus, we can save the parameters of the first image in the feature extraction path and avoid repeating computation. Then it uses final DMs drawn from each decision path to calculate the decision volume (DV), which records the activity levels of all the source images. The calculation process is acting as normalization to draw out relative clarity of each source images, which is shown below:

where $j\in\{1,..,N_{I}\}$, is the index of the source image and $p^{j}_{i}$ is the value of pixel $i$ in final DM when fuses the source image $1$ and the source image $j$.

Then, we choose the index of maximum in $DV^{j}_{i}$ for each pixel $i$ as the index of the most clarity pixel $i$ in the source images. According to the above indices, we can obtain the entire resulting fusion image.

It is important to notice that the decision calibration strategy can only applied to the DM based network structure without the empirical post-processing methods. Because those empirical post-processing methods, such as morphology operation and small region removal strategy, which used in CNN-Fuse (Liu et al. 2017) and SESF-Fuse (Ma et al. 2019), firstly require to convert the initial DM to the binary DM, which loss the relative clarity information and can not be used in the process of decision volume calculation. Our method, GACN, can draw out the decision map without the empirical post-processing methods, which is more suited to the decision calibration strategy in the application of multiple images fusion.

During training, all images were transformed to gray-scale and resized to $256\times 256$, then random cropped to $156\times 156$. Note that images were gray-scale in the training phase, while images for testing can be gray-scale or color images with RGB channels. For color images that needed to be fused, we transformed the images to gray-scale and calculated a decision map to fuse them. In addition, we used random crop, random blur, random offset, and gaussian noise as data augmentation methods. The initial learning rate was $1\times 10^{-4}$, and this was decreased by a factor of 0.8 at every two epochs. We optimized the objective function by Adam (Kingma and Ba 2015). The batch size and number of epochs were 16 and 50, respectively. Our implementation was derived from the publicly available Pytorch framework (Adam et al. 2019). The network’s training and testing were performed on a station using an NVIDIA Tesla V100 GPU with 32 GB memory.

We use six classical fusion metrics: $Q_{g}$ (Xydeas and Petrovic 2000), $Q_{y}$ (Yang et al. 2008), $Q_{ncie}$ (Qiang, Yi, and Jing 2008), $Q_{cb}$ (Chen and Blum 2009), $FMI\_EDGE$ and $FMI\_DCT$ (Haghighat, Aghagolzadeh, and Seyedarabi 2011) to evaluate the quality of fused result. $Q_{g}$ evaluates the amount of edge information transferred from input images to the fused image. $Q_{y}$ calculates the similarity between fused image and the sources(Yang et al. 2008). $Q_{ncie}$ measures the nonlinear correlation information entropy between the input images and the fused image (Qiang, Yi, and Jing 2008). $Q_{cb}$ is a perceptual quality measure for image fusion, which employs the major features in a human visual system model (Chen and Blum 2009). $FMI\_EDGE$ and $FMI\_DCT$ calculates the mutual information of the edge features and discrete cosine transform feature between the input images and the fused image (Haghighat, Aghagolzadeh, and Seyedarabi 2011). A larger value of any of the above six metrics indicates better fusion performance. For fair comparison, we use appropriate default parameters for these metrics, and all codes are derived from their public resources (Liu 2012; Mohammad and Amirkabiri 2014).

To demonstrate the performance of our method, we compare it with recent SOTA fusion methods in objective and subjective assessments.

We evaluate our method with different settings to verify the contribution of each module.

The example of multiple images fusion is shown in Table 3 and Figure 10. The microscopic image ’chip’ (with the size of $2700\times 1800$) was obtained by a microscope that took pictures with lots of different focus points. Decision calibration for ’chip’ images fusion can actually increase execution efficiency by about 30.65% compared to one-by-one serial strategy (0.7138’s to 0.4905’s for each image by using GACN), which is more feasible for industrial application. And the same increase of efficiency can also be found in CNN-Fuse and SESF-Fuse, which means that the decision calibration can be applied to other networks. Note that for decision calibration, we deleted the post-processing operations of DM in CNN-Fuse and SESF-Fuse for fair comparison.

The visualization of fusion result of GACN is more clear than that of CNN-Fuse and SESF-Fuse whether in decision calibration or serial strategy. In the future work, we try to eliminate the impact of defocused spread effect for multi-focus image fusion.