Dynamic Image Restoration and Fusion Based on Dynamic Degradation

Deep convolution neural network [3] has achieved remarkable results in many visual fields. However, the traditional convolution module often obtains static learning weights, which will not change in the test phase once the training is completed. This will severely limit the performance of the network model in changing scenarios. To solve this problem, the concept of dynamic convolution (e.g., CondConv [4], DynamicConv [5], and DyNet [6]) is proposed, i.e., the target convolution kernel is obtained by weighting a series of convolution kernels. Dynamic convolution has excellent performance in the task of classification and recognition but there is a lack of relevant research in the task of image fusion.

When existing image restoration [7] or super-resolution [8, 9] tasks generate degraded data, they often use limited degradation kernel (e.g., blurring, bicubicly downsampling, and white Gaussian noise) to simulate image degradation in a real environment. However, in this way, a single image will only be affected by a single blur kernel and downsampling, which is inconsistent with the real environment, i.e., a single image will be affected by a variety of degradation kernels. Image degradation is of great research value not only for image restoration task or image super-resolution task, but also for image fusion task, especially for cross-modal image fusion task without ground truth. After all, image fusion is to improve the real quality of the image, not the similarity of original images. Unfortunately, no relevant research has been found in the field of cross-modal image fusion.

Due to the outstanding performance of deep learning methods in many visual fields, deep-learning-based image fusion methods have been widely proposed,e.g., DenseFuse [10], IFCNN [11], FusionDN [12], PGMI [13], U2Fusion [14], NestFuse [15], RFN-Nest [16]. Existing deep-learning-based image fusion networks, e.g., image fusion methods based on Siamese networks and generative adversarial networks , often use similarity loss (e.g., content loss, detail loss, gradient loss, and tee loss) or adversarial loss to learn static fusion weight. For the cross-modal image fusion task without ground truth, it is necessary to calculate the similarity between the predicted image and the original image, so as to retain the texture details of source images as much as possible. Li et al. [10] introduced a SSIM [17] to optimize the image fusion weight. Many cross-modal image fusion methods (e.g., IFCNN, FusionDN, NestFuse, and U2Fusion) based on deep learning use this similarity measure optimization function, e.g., SSIM, MSE, and content loss. However, Ma et al. [18] introduced an adversarial loss into the infrared and visible image fusion task. Although the introduction of adversarial loss provides a new idea for image fusion, the subjective fusion quality obtained by adversarial loss has an obvious local blur effect. Therefore, to solve this problem, Ma et al. [19] introduced a dual discrimination generative adversarial fusion network that combined multiple loss functions (e.g., content loss, detail loss, gradient loss, tee loss, and adversarial loss) are used to optimize the fusion weight. In addition, the fusion criterion of existing image fusion methods is usually defined as the fusion layer after the feature extraction layer, which separates the fusion from the subsequent decoding convolution. Although this method is widely used, there are also some irrationality. 1) The fusion criterion of artificial design needs specific expert experience. 2) Artificially designed fusion standards will limit the versatility of the network, such as infrared and visible light, multi-focus image fusion tasks. To solve above problems, we carried out the related research of dynamic image fusion and proposed the dynamic degradation for image restoration and fusion.

The robustness of image fusion is of great significance for its application and development. The existing image fusion methods lack research on dynamic fusion weight and dynamic degradation model, i.e., the visual data is affected by illumination, blur, and compression transmission at the same time, which is dynamic and coupled with each other, as verified in section 4. The existing image fusion methods do not consider this dynamic degradation problem in the feature extraction stage. In the dynamic visual scene, the original feature extraction weight will be invalid, and the image fusion effect will be seriously affected. Although there is comprehensive research on the degenerate kernels in the existing image restoration tasks, only the influence of a single degenerate kernel (e.g., downsampling combined with blur kernel) on the overall visual scene is considered. However, dynamic fusion weight and dynamic degradation models play an important role in improving the robustness of image fusion. Therefore, we are motivated to study dynamic fusion and dynamic degradation models in the field of image fusion.

Image restoration and fusion are not only to calculate the similarity between the predicted image and different modal images but also to restore high-quality image. Although existing image restoration methods have done a lot of research on different degradation kernels, existing degradation kernels are mostly static degradation kernels, only considering the global degradation of the image (i.e., a single degradation kernel is used for the whole image), without considering the local degradation of the image (i.e., a single image will be affected by different degradation models). The traditional degradation model H on Z (high quality image) is defined as [9]:

where $\bm{k}$ indicates a 2-D convolution kernel, denotes convolution operation, ↓s represents down-sampling process with a scaling factor s, N indicates white gaussian noise. The existing degradation convolution $\bm{k}$ is a single static convolution kernel, which cannot simulate the degradation of real images. Therefore, in order to better simulate the real degradation model, we define $\bm{k}$ as a linear combination of a series of basic degradation kernels, i.e., isotropic Gaussian kernel, motion blur kernel, and anisotropic Gaussian kernel. The dynamic kernel is defined as:

where a, b, c indicate the weight of basic degradation kernel, $k_{m}(\bm{j})$ denotes motion blur kernel, $k_{i}(\bm{j})$ indicates isotropic Gaussian kernel, $k_{a}(\bm{j})$ indicate anisotropic Gaussian kernel, $x$ respresents a random integer variable. In order to avoid the image oscillation caused by the dynamic degradation kernel, the weights $a,b,c$ must be added to sum to 1. Therefore, the dynamic degradation model $H_{m}$ can be formulated as:

For the cross-modal image fusion task, taking the infrared and visible image fusion task as an example, we need to restore three dynamic degradation models, i.e., visible image $(H_{1})$, infrared image $(H_{2})$, and fused image $(H_{3})$.

Existing image fusion methods often use artificial fusion criteria in feature space or learn fixed fusion weights by the data-driven method. The fusion criterion for artificial design is defined as:

where $\bm{f}_{i}$ and $\bm{f}_{v}$ denote infrared and visible feature maps; $w_{i}(\bm{x},\bm{y})$ denotes the fusion weight of the position of feature map $(\bm{x},\bm{y})$; $ø$ indicates the fused feature maps; $C(*)$ denotes concatenate operation; $ø$ indicates fused feature maps. This method focuses on the design and improvement of the fusion weight strategy of different feature maps, e.g., the fusion weight based on saliency, weighted average, maximum, and sum. For static CNN, the fusion weight can be formulated as:

where $W^{T}$ and $b$ denote the convolution process. To overcome the shortcomings of the existing fusion methods (i.e., artificial fusion criterion and static fusion weight), we propose a dynamic fusion method in kernel space. We concatenate the convolution kernel parameters learned by the feature extraction module, and linearly stack the convolution kernel weights combined with the channel attention mechanism. The dynamic fusion criteria is defined as:

where $W_{k}$ denotes k-th convolution kernel; $N$ the number of convolution kernel of dynamic kernel; $\pi_{k}(\bm{x})$ denotes convolution kernel weight, i.e., channel attention weight; $C(\bm{f}_{i},\bm{f}_{v},\bm{k}_{di},\bm{k}_{dv})$ denotes the dynamic degenerate kernel (i.e., $\bm{k}_{di}$, and $\bm{k}_{dv}$) is superimposed on the feature maps, i.e., visible feature map ($\bm{f}_{v}$), and infrared feature map ($\bm{f}_{i}$) in the channel dimension.

Existing deep-learning-based image fusion methods mainly use similarity measure loss to optimize image fusion weight, and the similarity fusion loss ${\cal L(\bm{\theta})}$ is defined as:

where $\bm{X_{i}}$ indicates i-th image to be fused; $\bm{\theta}$ denotes CNN’s weight; $\bm{P}$ indicates fused image; $S(*)$ denotes similarity function. For the cross-modal image fusion task, due to the limitations (i.e., just represents similarity, rather than high-quality image) of the existing similarity loss, this paper introduces the idea of adversarial learning based on the dynamic degradation model and proposes the dynamic negative sample fusion loss ${\cal L(\bm{H}_{i})}$ that is defined as:

where $\bm{H}_{i}$ denotes i-th dynamic model. The basis of adversarial learning is to have positive samples as guidance so that the network model can be optimized in the right direction. Therefore, we introduce a positive sample loss ${\cal L(\bm{\theta})}$, which is defined as:

where $M_{i}$ denotes i-th image fusion method; $O(*)$ indicates evaluation function. Therefore our fusion loss function L is defined as:

Our fusion loss considers not only the similarity with the original image, but also the negative sample distance based on the dynamic degradation model. This method can effectively reduce the universality of image fusion caused by the difference of data distribution.

Our network framework adopts the classic Siamese network framework. In the two-way feature extraction branch network, the network weights are not shared. Traditional image fusion methods based on Siamese network or self-coding network, e.g., Deepfuse, Densefuse, and IFCNN, two sub-modules are purely for extracting features from different images. Different from the traditional image fusion method based on Siamese networks or self-coding network, our feature extraction branch is not only to extract features but also to restore the dynamic degradation model. Notably, the most important thing is that our network does not need datasets from other fields to assist learning. i.e., the cross-modal image fusion task completes the restoration of the dynamic degradation model with its own data. This strategy effectively reduces the complexity of training and avoids the problem of model recovery caused by the difference in data distribution. Different from the existing deep-learning-based methods (e.g., NestFuse, RFN-Nest, and SAF), our network does not need complex network structure, e.g., RESNET, NestNet, and DenseNet. Our fusion module uses the superposition of four layers of dynamic convolution, and the weight of the convolution kernel is generated by the channel attention mechanism. In the process of training, our dynamic degradation kernel will be serialized, dimension reduced and stretched to align with different modal data in dimension. When the same image is affected by multiple degradation kernels, it is difficult for a simple single branch network to converge. That is reason why we avoid a data-driven training model in feature space.

In this section, datasets, metrics and methods for experimental evaluation are first presented. Then, implementation details of evaluated methods are introduced.

1) Datasets: We carry out experiments on four datasets, i.e., FLIR, RGB-t, gun, and TNO.

a) FLIR [26]: FLIR dataset is obtained by RGB and thermal imaging camera installed on the vehicle and contains 14, 452 thermal infrared images, including 10, 228 from short video and 4, 224 from the 144-second video. Unfortunately, there is no registration.

b) RGB-T [27]: The RGBT dataset includes 821 image pairs.

c) Gun [11, 28]: Gun is captured in night mode contaminated with blur and noise. There is a lot of duplication between this dataset and the TNO dataset. Therefore, we only use one image of $gunA\&gunB$ in this dataset.

d) TNO [29]: It contains multi-spectral (near-infrared and long-wave infrared or thermal) night images of different military-related scenes, registered in different multi-band camera systems. There are 21 pairs of images commonly used in existing image fusion methods.

2) Metrics: The distinctiveness of an image quality is usually quantitatively evaluated using entropy (EN) [20], average gradient (AG) [21], structural similarity (SSIM) [17], visual information fidelity (VIF) [22], natural image quality evaluator (NIQE) [30], PSNR [23], are defined as 1.

3) Methods: As shown in Table 2, we compared 7 state-of-the-art methods, i.e., 5 deep-learning-based methods, 1 GAN method, and 1 traditional method, published in the top journal.

4) Implementation details: Before the experiment, we need to clarify the following questions. All the experimental data were tested in the same environment. Our experimental platform is desktop 3.0 GHZ i5-8500, RTX2070, 32G memory.

In order to highlight the advantages of this method, we carried out comparative experiments and analytical experiments on the task of cross-modal image fusion and image restoration.

In this section, we analyze the impact of static and dynamic on image restoration and image fusion tasks.