Local Contrast and Global Contextual Information Make Infrared Small Object Salient Again

There are two main types of ISOS methods, traditional methods based on mathematical modeling and deep learning methods based on neural networks.

Among the traditional methods, Max-mean-Max-medium[12] and Top-Hat[42] used filters to separate target of background. LCM[3], ILCM[17], TLLCM[16] and MPCM[39] segmented small object by designing salient measures. IPI model[14] treated the input as a superposition of low-rank background and sparse yet shaped target, and solved such issues by using Low-rank decomposition, further methods like sound low-rank[28] and prior constraints[35] were proposed based on IPI. These methods suffered from low performance under complex scenarios.

As for deep learning methods, Wang et al. [36] used conditional GAN[26] with two generators and one discriminator to gain a great balance between miss detection and false alarm (MDvsFA). Dai et al. [10] proposed an asymmetric contextual modulation to help network performance well and introduced the first public ISOS dataset SIRST in real scenes, Dai et al. [11] further applied a handcraft dilated local contrast measure into network. Liu et al. [21] firstly introduced multi-head self-attention into ISOS tasks and got a good result. Zhang et al. [46] proposed AGPCNet with attention-guided context block and context pyramid module. Zhang et al. [45] took shape into account and designed Taylor finite difference edge block and two-orientation attention, and also proposed a more challengable dataset IRSTD. Deep learning methods have become more and more dominant in ISOS due to their great ability of robustness and generalization.

The ordinary convolution operation is to divide the feature map into patches of the same size as the convolution kernel and then perform a weighted sum operation, this fixed operation at each position may be suboptimal for some specific tasks. Therefore, researchers have proposed some advanced convolution operators. In order to solve the problem of standard convolution treating all input pixels as valid ones in image inpainting tasks, Liu et al. [22] proposed partial convolution, where the convolution is masked and renormalized to be conditioned on only valid pixels. Yu et al. [40] proposed gated convolution to provide a learnable dynamic feature selection mechanism for each channel at each spatial location across all layers. While in modeling geometric transformations, vanilla convolution is limited due to its fixed geometric structure, Dai et al. [6] proposed deformable convolution to enhance this ability by adding additional offsets learned from target task. Zhu et al. [48] then proposed deformable convolution v2 to reduce the impact of irrelevant region by adding additional weight and mimicking features from RCNN[29]. Yu et al. [41] proposed central difference convolution in face anti-spoofing task, which is able to capture intrinsic detailed patterns via aggregating both intensity and gradient information.

One common concern is the ability of the network to grasp the local and global information, local context usually are easy to extract using convolution, while how global the information can get usually determined by the receptive fields of the network. The most common strategy to enlarge the receptive fields is stacking convolutions and downsamplings constantly, continuous convolution can linearly enlarge receptive fields while downsamplings multiply, dilated (atrous) convolution inserts holes between pixels in convolutional kernels and can obtain larger receptive fields than standard convolution, Chen et al. [4] proposed dilation atrous pyramid pooling (ASPP) to capture multi-scale information and Wang et al. [37] designed a hybrid dilated convolution to enlarge receptive fields, they both got good improvements in semantic segmentation tasks. Attention mechanism [38, 34] can gain global information by calculating the correlation of each single pixel with other pixels. Fu et al. proposed spatial attention and channel attention [13] and achieved great results. Chi et al. [5] proposed fast Fourier convolution which performs convolution operators in frequency domain to conduct a global influence in spatial domain thus it can extract image-level information. Suvorov et al. proposed LAMA[33] which successfully applied it in large kernel image inpainting tasks. Berenguel et al. [2] then used FFC in monocular depth estimation and semantic segmentation.

We draw inspiration from a series of experiments using common segmentation networks, including FPN[20], U-Net[30], PSPNet[47] and DeepLabv3[4], the results in Table 1 indicate two anomalies: first, the performance is quite polarized, with the models that fuse more low-level features (FPN and U-Net) significantly outperforming the others; second, their performance drops only slightly as the network width and depth increase.

We further visualize the attention maps of 4 stages using grad-cam[31, 15] in FPN. As we can see in Fig. 2, the small object has been completely lost due to the excessive downsampling. Meanwhile directly increasing the width and depth of network still fails to mine more information while may lead to redundancy of parameters. Based on these observation, we summarize two essential guidelines on ISOS: 1) Extracting global information while maintaining high resolution may avoid small object being overwhelmed. 2) Extracting additional essential information using modules dedicated to ISOS could be more effective than directly increasing the depth and width of a network.

Based on the enlightenment from Sec. 3.1, we proposed a U-shape network with central difference convolution (CDC) and fast Fourier convolution (FFC). The whole framework can been seen in Fig. 3. The number of base channels is 32 and only 4 downsampling operations are performed in the whole process, which satisfies the needs of ISOS tasks while avoiding inefficient redundancy. Specially, during the downsample process, we use central difference convolution residual block which contains a standard convolution and CDC with a residual convolution for channel alignment. We use two cascaded FFC to form our fast Fourier convolution residual block which aims to extract global context in high resolution. In what follows, we will briefly introduce central difference convolution in Sec. 3.3 and fast Fourier convolution in Sec. 3.4.

Convolution can effectively extract color, shape, texture and other information, while standard convolution may be limited because the lack of these information in ISOS tasks. Inspired by human visual system sensitive to intensity difference and contrast, we utilize central difference convolution to guide our network. CDC can introduce additional contrast information by computing the difference between the center point and other points within the convolution window and can be described as Eq. 1,

$\displaystyle f(x,y)=\sum_{(i,j)\in R}w_{(i,j)}\cdot(F_{(x+i,y+j)}-F_{(x,y)})$

(1)

In addition, CDC is often combined with vanilla convolution to retain some of the traditional feature extraction capabilities, and the whole process can be expressed in the equation.

	$\displaystyle CDC(x,y)=$	$\displaystyle\theta\cdot\sum_{(i,j)\in R}w_{(i,j)}(F_{(x+i,y+j)}-F_{(x,y)})$
		$\displaystyle+(1-\theta)\cdot\sum_{(i,j)\in R}w_{(i,j)}(F_{(x+i,y+j)})$		(2)

where the hyperparameter $\theta\in(0,1)$ is used to determine the contribution between CDC and vanilla convolution. The window size $R$ used to calculate the difference is equal to the convolution kernel size while the receptive fields of it can naturally increase during the forward process, thus making it capable of extracting multi-level contrast information which further help the network identify infrared small objects with different size and effectively guide the network performs well.

Another issue of ISOS is that the small objects are easily overwhelmed in the excessive downsample layers which is used to obtain global information. In order to solve this contradiction, rather than using the Transformer model with long range dependency, we employ fast Fourier convolution (FFC) which can gain image-level receptive fields in high resolution thus we can extract global context without losing small objects. We follow the structure in LAMA[33] for FFC. Specifically, we splits the channels into two parallel branch, the local branch uses standard convolution to extract local information while global branch applies the Fourier Unit(FU) to gain global context, and there are two additional short-cuts for information fusion. The whole process can described as:

	$\displaystyle Y_{l}=Conv_{l->l}(x_{l})+Conv_{g->l}(x_{l})$		(3)
	$\displaystyle Y_{g}=Conv_{l->g}(x_{g})+FU_{g->g}(x_{g})$		(4)

The Four Unit is key to get global context, because it transforms input features from spatial domain to frequency domain where each single point corresponds to all points in the spatial domain. Therefore, the convolution operation being conducted within a small kernel size is able to influence the all image in spatial domain and obtain global contextual information. The FU makes following steps:

1. 1)

   Transform the input feature map from spatial domain to frequency domain with Real FFT2d and concatenates the real and imaginary parts

   	RealFFT2d	$\displaystyle:\mathbb{R}^{H\times W\times C}\rightarrow\mathbb{C}^{H\times\frac{W}{2}\times C}$
   	ComplexToReal	$\displaystyle:\mathbb{C}^{H\times\frac{W}{2}\times C}\rightarrow\mathbb{R}^{H\times\frac{W}{2}\times 2C}$

2. 2)

   Apply convolution, normalization and activation function in the frequency domain

   $$Conv\circ Norm\circ Act:\mathbb{R}^{H\times\frac{W}{2}\times 2C}\rightarrow\mathbb{R}^{H\times\frac{W}{2}\times 2C}$$

3. 3)

   Transform inversely from frequency domain to spatial domain

   	RealToComplex	$\displaystyle:\mathbb{R}^{H\times\frac{W}{2}\times 2C}\rightarrow\mathbb{C}^{H\times\frac{W}{2}\times C}$
   	Inverse RealFFT2d	$\displaystyle:\mathbb{C}^{H\times\frac{W}{2}\times C}\rightarrow\mathbb{R}^{H\times W\times C}$

We concatenate the local branch and global branch into one in the end, and it contains rich local and global information.

Datasets. We evaluate our methods on two widely-used datasets in ISOS: SIRST[10] and IRSTD[45]. SIRST contains 427 images in real IR scenes with half of the objects in SIRST only contains $0.1\%$ pixels of whole image. Larger than SIRST, IRSTD contains 1001 images with more challenging object and complex backgrounds. Both SIRST and IRSTD are separated into training set and test set with a ratio of 8:2.

Metrics. Following privious works, we use pixel-level metrics (IoU and nIoU) and target-level metrics (Pd and Fa) to measure our method. Intersection over Union (IoU) and normalized Intersection over Union (nIoU) can described as:

	$\displaystyle IoU$	$\displaystyle=\frac{1}{n}\cdot\frac{\sum_{i=0}^{n}tp_{i}}{\sum_{i=0}^{n}(fp_{i}+fn_{i}-tp_{i})}$		(5)
	$\displaystyle nIoU$	$\displaystyle=\frac{1}{n}\cdot\sum_{i=0}^{n}\frac{tp_{i}}{fp_{i}+fn_{i}-tp_{i}}$		(6)

Where $n$ means to the total number of samples, $tp$ denotes the true positive, $fp$ denotes false positive and $fn$ denotes false negative. While target-level metrics probability of detection (Pd) and false alarm rate (Fa) can be described as:

	$\displaystyle Pd$	$\displaystyle=\frac{1}{n}\cdot\sum_{i=0}^{n}\frac{N_{pred}^{i}}{N_{all}^{i}}$		(7)
	$\displaystyle Fa$	$\displaystyle=\frac{1}{n}\cdot\sum_{i=0}^{n}\frac{P_{false}^{i}}{P_{all}^{i}}$		(8)

where $N_{pred}$, $N_{all}$ denotes the number of correct detected objects and the number of total objects and $P_{false}$, $P_{all}$ denotes the pixels of false detected objects and the pixels of total objects. We regard that the detection is correct when the distance between the centers of the predict result and the ground truth is less than 4.

We conduct experiments on a computer with a 2.50GHz CPU, 16GB RAM and GeForce RTX 3090 based on Pytorch. For more details, we use AdamW optimizer with an initial learning rate of 0.001 and decayed by Cosine-Anneling-LR[25] schduler and we use binary cross entropy loss and soft IoU loss as our criterion. Each experiment is trained for 300 epochs with a batch size of 8. Our UCF achieves the best performance with a CDC ratio $\theta$ of 0.7 while using 7 FFC residual blocks.

We evaluated several traditional, deep learning for ISOS and common segmentation methods and compared their results, which are shown in Table 2. Overall, deep learning methods outperformed traditional ones due to their powerful feature extraction and generalization capabilities. Our proposed method (UCF) demonstrated superior performance over other deep learning methods on both datasets, as evidenced by all metrics. Notably, on SIRST, UCF achieved an impressive performance of 80.89 IoU and 78.72 nIoU, representing an great improvement over other methods, while also achieving a perfect detection rate and an extremely low false alarm rate of only $2.22\times 10^{-6}$. On IRSTD, our method also outperformed state-of-the-art deep learning methods by 3-8 points in IoU and nIoU, achieving scores of 68.92 and 69.26, respectively.

We conduct a further investigation of the dynamic relationship between Precision and Recall using LCM[3], IPI[14], ACM[10], AGCP[46], and our proposed method UCF. The ROC curves are presented in Fig. 4, where the area under the curve (AUC) is a key metric for quantitatively evaluating the ROC. Our UCF method achieved the highest AUC and F-score on both datasets, as shown in Table 3.

Several visualization results for different methods are presented in Fig. 5. These results clearly demonstrate that our proposed UCF method not only achieves a higher detection rate and fewer false alarms at the object level, but also predicts object shapes more accurately.

We conduct a series of ablation experiments on SIRST to investigate the effectiveness of CDC and FFC. In Table 4, we demonstrate the general effectiveness of these methods. CDC improves performance by incorporating local contrast information, while FFC provides valuable global information. When combined, CDC and FFC achieve the best results in terms of IoU, nIoU, and Pd. However, the Fa score drops slightly compared to using only FFC. This is because CDC tends to fix shape and texture details, which can sometimes result in false pixels.

Effectiveness of CDC.

We conduct experiments with different hyperparameter $\theta$ which determines the ratio between CDC and vanilla convolution. As shown in Fig. 6(a), CDC consistently outperforms vanilla covolution by mining essential contrast information and we achieve the best performance when $\theta=0.7$. In addtition, we compare the performance of deformable convolution and gated convolution with that of CDC. As show in Table 5, gated convolution only has a slight improvement over vanilla convolution, while deformable convolution has a significant drop a lot, this is because the sharp information is quite insufficient in ISOS tasks and deformable convolution fails to learn the offsets according to the target’s shape. These results indicate that our proposed CDC has clear advantages over other convolution operators for ISOS.

Effectiveness of FFC. In our FFC block, we conduct experiments to study the inner parameter of $n$ which determines the number of FFC blocks used in our method. As shown in Fig. 6(b), the FFC residual block, designed to extract global context, significantly improves performance, and we achieve the best results when using five FFC residual blocks. We further explore other methods for extracting global information, such as dilated convolution and double attention, as mentioned in Sec. 2.3, from Table 6 we can see that dilated convolution and double attention both show improvements in performance by enlarging receptive fields and extracting global context. However, dilated convolution only captures image-level information in deep layers, while attention mechanisms lack local inductive bias. Therefore, they are inferior to FFC in ISOS tasks.