Infrared and Visible Image Fusion with Hierarchical Human Perception

Traditional methods adopt sparse representation [12] and multi-scale transform [13] to handle information extraction and fusion. DenseFuse[3] is the first deep learning (DL)–based infrared and visible image fusion method. FusionGAN[14] utilizes generative adversarial network (GAN) to integrate thermal radiation and texture in an implicit way. TarDAL[15] cascade image fusion and object detection to mine information beneficial for both human inspection and high-level tasks. DDFM[16] formulates fusion task as a conditional generative problem under the denoising diffusion model. FILM[17] generates textual descriptions via the ChatGPT to convey a deep semantic understanding of the fusion network, guiding the extraction of crucial visual features.

CLIP[11] learns from natural language supervision at large-scale tasks during pre-training, exhibiting robust transfer performance without fine-tuning on various tasks, which have been widely used in low-level vision tasks. BLIP[18] bootstrap image captions from noisy web data, which performs substantial improvement on different vision-language tasks. LLaVA[10] is an Large Multimodal Model (LLM) that connects visual encoder of CLIP with the large language models, and fine-tunes the model on their generated instructional vision-language data. It can be capable of multimodal chat abilities for visual and language understanding.

Infrared images can highlight the thermal targets and provide high contrast with the background where people can distinguish essential information. Visible images usually contain finer details, such as texts and signs, where people also tend to pay attention to. Meanwhile, the overall information describing an image is also mindful.

We have devised four questions from a global context to specific local regions, in order to simulate human curiosity and concerns when observing image pairs that are abundant in both thermal and detailed information. For example, when we view the infrared and visible image pair in Fig. 1, we first attempt to comprehend the overall content so we ask LLaVA to describe the image (Q4). Subsequently, given that infrared images can usually highlight thermal targets, our objective is to identify the presence of various objects within the image (Q3). Lastly, infrared images have higher contrast than visible images, but visible images can also contain more details within a specific region. Therefore, we will focus on specific regions with high contrast and rich information content (Q1 and Q2). The generated textual prompts which incorporate hierarchical human priors are fused with the fusion network to guide the retention of useful visual features.

The overall architecture is shown in Fig. 2. For the given infrared and visible image $I_{ir}$ and $I_{vis}$, we design four questions that people tend to notice according to the thermal information of infrared and texture information of visible images to simulate hierarchical human perception.

Human Perception Module (HPM) consists of LLaVA and CLIP encoder to generate text features $\Phi_{ir}^{T}$ and $\Phi_{vis}^{T}$ as human priors. To guide the fusion of visual features, we use $1\times 1$ convolutional layer to reduce the text features of the 4 answers to 1 dimension, then concatenate them to form the fused text feature $\Phi_{i-s}^{T}$.

The architecture of the fusion network follows the previous work MDA[19] as baseline, and the fused text feature $\Phi_{i-s}^{T}$ is combined with visual feature $M_{ir}$ and $M_{vis}$ extracted by the convolutional block of MDA, guiding the subsequent fusion of visual features, which is as follows:

where $CA(\cdot)$ denotes the cross-attention mechanism, and the Query (Q) is calculated by the $\Phi_{i-s}^{T}$, while Key (K) and Value (V) are calculated by the $M_{ir}$ and $M_{vis}$. We cascade two Cross-attention Blocks to integrate the text and visual features comprehensively.

Finally, $F_{ir}$ and $F_{vis}$ are fused with the multi-scale encoder and decoder of MDA and then reconstruct the fused image $I_{f}$.

The architecture of HPM is shown in Fig. 3. For the input image pair, we ask four questions and answer via LLaVA according to infrared and visible images. After obtaining the text of answers $T_{ir}\in\mathbb{R}^{77\times 4}$ and $T_{vis}\in\mathbb{R}^{77\times 4}$ from LLaVA, we encode texts of these answers by parameter-frozen CLIP text encoder $\mathcal{E}_{text}$ to get the text features $\Phi_{ir}^{T}$ and $\Phi_{vis}^{T}$, guiding both fusion of visual features and optimization of training process.

To guide the fusion of visual features, $\Phi_{ir}^{T}$ and $\Phi_{vis}^{T}$ are dimensionally reduced through the $1\times 1$ convolutional layer and concatenation, and then $\Phi_{i-s}^{T}$ is fused with visual features $M_{ir}$ and $M_{vis}$ by cross-attention mechanism. To guide the optimization of training process, the input image pair $I_{ir}$ and $I_{vis}$ are encoded by the parameter-frozen CLIP image encoder $\mathcal{E}_{img}$ to generate visual features $\Phi_{ir}^{V}$ and $\Phi_{vis}^{V}$, and text-image similarity inner the batch is calculated by the cosine similarity between $\Phi_{ir/vis}^{V}$ and $\Phi_{ir/vis}^{T}$ to represent semantic distribution.

Similarly, text-image similarity inner the batch of the fused image $I_{f}$ is calculated via the above method as well. Specifically, after obtaining the fused image $I_{f}$, we also ask the same four questions for $I_{f}$ and use LLaVA to generate the text of answers $T_{f}$, and encode answers into text features $\Phi_{f}^{T}$ by CLIP text encoder $\mathcal{E}_{text}$. Then the cosine similarity can be calculated by the $\Phi_{f}^{T}$ and $\Phi_{f}^{V}$.

The degree of information retention is determined by the optimization objective, thus we construct image loss $L_{image}$ to preserve both intensity and detail information from source images, while hierarchical semantic loss $L_{hier}$ aims at driving texts of answers guiding the preservation of information that prioritizes human focus and comprehension.

The image loss includes pixel intensity and detail items to maintain informational fidelity, which are maximum intensity loss, maximum gradient loss and structural similarity index metric (SSIM) loss, as shown in (2) and (3).

The hierarchical semantic loss constrains the salient regions that humans focus on between fused image and source image pair. We expect the text-image similarity between the fused image and source image pair to be close, indicating that the distribution of each image with its textual answers is similar between the fused image and source image pair. Firstly, we measure the similarity score between the text vector of an answer and the corresponding image vector by calculating the cosine similarity in the CLIP space, which is as follows:

where $m\in\{ir,vis,fusion\}$, and $i$ denotes other samples within a batch. $\mathcal{E}_{img}$ and $\mathcal{E}_{text}$ are parameter-frozen CLIP image and text encoder, respectively. Then, we adopt the similarity score between the fused image and the source image pair to be closer for the four answers, which expressed as hierarchical human concerns:

where $j$ corresponds to four different question-answer sets.

The total loss function is composed of pixel intensity loss, detail loss and hierarchical semantic loss, which is as follows:

We set values of $\alpha$ and $\beta$ to 4 and 1, respectively.

We choose six state-of-the-art image fusion methods to compare the performance of preserving salient targets and detail information, which are RFN-Nest[20], LRRNet[21], YDTR[22], GANMcC[23], EMMA[24] and our baseline MDA[19]. Qualitative results are shown in Fig. 4. The first two columns are source infrared and visible images, and the red and blue boxes in the fusion results from the third to ninth columns enlarge the details in the fused images. Some thermal targets exhibit low intensity in visible images, such as humans in the second rows and vehicles in the last rows. Compared with LRRNet and YDTR, our HPFusion can highlight the saliency of these targets, meanwhile keeping the contrast of visible images. Additionally, the detail information is effectively preserved in our results, like the sharp edges of targets and background, which is damaged in fused images generated by the GANMcC.

Five quantitative metrics are employed for evaluation, which are mean squared error (MSE), structural similarity index metric (SSIM), correlation coefficient (CC), peak signal-to-noise ratio (PSNR) and edge retentiveness ($Q^{AB/F}$). These metrics provide a comprehensive assessment of the retention of pixel intensity and edge information transferred from the source images. Specifically, a lower MSE indicates a higher degree of preserving pixel intensity, whereas the other four metrics indicate the opposite trend. Comparison results are shown in upper part of Table I, where the method proposed achieves the best performance in four metrics, indicating the effective preservation of thermal and detail information.

We investigate the effectiveness of HPM and hierarchical semantic loss in our ablation studies. Quantitative results are shown in the lower part of Table I. Our HPFusion consisting of HPM and $L_{hier}$ achieves best performance across four metrics, which demonstrates the superiority of HPFusion in balancing the thermal saliency and texture details.