Diffusion Model for Camouflaged Object Detection

Existing COD methods [11, 12, 13] are based on a non-generative approach to segment the objects from the background. The approaches in COD can be broadly categorized into the following strategies: a) Introducing additional cues to facilitate the exploration of camouflage features. BGNet [50] uses edge semantic information to enable the model to extract features that highlight the structure of the object and thus pinpoint the object boundary. TINet [65] designs a texture label to find boundaries and texture differences through progressive interactive guidance. FDCOD [63] incorporates frequency domain features into CNN models to better detect objects from the background. DGNet [24] utilizes gradient edge information to facilitate the generation of contextual and texture features. b) Multi-task learning strategies are used to improve segmentation capabilities. ANet [31] proposed joint learning of classification and segmentation tasks to help the model improve recognition accuracy. UJSC [32] detects both salient and camouflaged objects to improve the model performance. Rank-Net [36] proposes to use the localization model to find the obvious discriminative region of the camouflaged object, and the segmentation model to segment the full range of the camouflaged object. c) Coarse-to-fine feature learning strategy is utilized to explore and integrate multi-scale features. SegMaR [27] uses multi-stage detection to focus on the region where the goal is located. ZoomNet [40] learns multi-scale semantic information through multi-scale integration and hierarchical hybrid strategies to promote models that produce predictions with higher confidence. PreyNet [61] imitates the predation process for stepwise aggregation and calibration of features. PFNet [37] mimics nature’s predation process by first locating potential targets from a global perspective and then gradually refining the fuzzy regions. SINet [13] is designed to improve segmentation performance by locating the object first and then differentiating the details. $C^{2}$FNet [49] proposes to use global contextual information to fuse on high-level features in a cascading manner to obtain better performance. HitNet [22] and FSPNet [23] propose to explore global context cues by transformers. In this paper, we introduce generative models, i.e., denoising diffusion models, into the COD task to gradually refine the object masks from the noisy image, which achieve excellent performance, especially for objects with fine textures.

The diffusion model [20, 47] is a generative model that uses a forward Gaussian diffusion process to sample a noisy image, and then iteratively refines it using a backward generative process to obtain a denoised image. Diffusion models have shown strong potential in several fields, such as image synthesis [10, 20], image editing [19], and image super-resolution [9]. Moreover, the learning process of diffusion models is able to capture high-level semantic information that is valuable for segmentation tasks [3], which has led to a growing interest in diffusion models for image segmentation including medical image segmentation [53, 54], semantic segmentation [4, 26, 55, 57], and instance segmentation [1, 17]. MedSegDiff [53] proposes the first DPM-based medical segmentation model, and MedSegDiff-V2 [54] further improves the performance based on it using transformer. DDeP [4] finds that pre-training a semantic segmentation model as a denoising self-encoder is beneficial for performance improvement. DDP [26] designs a dense prediction framework with stepwise denoising refinement guided by image features. ODISE [57] combines a trained text image diffusion model with a discriminative model to achieve open-vocabulary panoptic segmentation. DiffuMask [55] uses a model for the automatic generation of image and pixel-level semantic annotations, and it also shows superiority in open vocabulary segmentation. DiffusionInst [17] proposes the first instance segmentation model based on a diffusion process to achieve global instance mask reconstruction. Segdiff [1] uses a diffusion probabilistic approach to design an end-to-end segmentation model that does not rely on a pre-trained backbone. However, there are no studies that demonstrate the effectiveness of diffusion models in COD tasks. In this work, we present the first diffusion model for the COD segmentation task.

The diffusion probability model has reaped plenty of attention due to its simple training process and excellent performance. It is mainly divided into forward process and reverse process. In the forward process, noise is added to the target image to make it closer to the Gaussian distribution. The reverse process learns to map the noise to the real image.

The forward process refers to the gradual incorporation of Gaussian noise with variance $\beta_{t}\in(0,1)$ into the original image $x_{0}\sim p\left(x_{0}\right)$ at time $t$ until it converges to isotropic Gaussian distribution. The forward process is described by the formulation:

where $t\in[1,T]$. We can obtain the latent variable $x_{t}$ directly by using $x_{0}$ by the following equation:

where $\alpha_{t}:=1-\beta_{t}$, $\bar{\alpha}_{t}:=\prod_{s=0}^{t}\alpha_{s}$ and $\epsilon\sim\mathcal{N}(0,\mathbf{I})$.

The reverse process converts the latent variable distribution $p(x_{T})$ to $p(x_{0})$ through a Markov chain, and the reverse process can be denoted as follows:

The combination of $q$ and $p$ is a variational auto-encoder, and the variational lower bound (VLB) is defined as follows:

As shown in Figure 2, the proposed diffCOD aims to solve the COD task by the diffusion model. The denoising network of diffCOD is based on the UNet architecture [44]. To get effective conditional semantic features, we obtain multi-scale features by ViT-based backbone and feature fusion (FF) to yield features containing rich multi-scale details. In addition, to let the texture patterns and localization information in the conditional semantic features guide the denoising process, we propose an injection attention module (IAM) based on cross-attention. This allows the network to reduce the difference between diffusion features and image features and to combine the advantages of both.

Thus the IAM operation is defined as follows:

where $\mathbf{M}^{att}_{1}$ and $\mathbf{M}^{att}_{2}$ represent the attention maps of $\mathbf{Q^{D}}$-$\mathbf{P^{F}}$ and $\mathbf{K^{D}}$-$\mathbf{P^{F}}$, respectively. $O^{I}\in\mathbb{R}^{\frac{H}{32}\times\frac{W}{32}\times C}$ denotes the final generated cross-attention fusion feature.

In the forward process, the Gaussian noise $\epsilon_{t}$ is added to the ground truth $y_{0}$ to obtain the noise mapping ${y}_{t}\sim q\left({y}_{t}\mid{y}_{0}\right)$ by $T$-steps. The intensity of the noise is controlled by $\alpha_{t}$ and conforms to the standard normal distribution. This process can be defined as follows:

where $t=[1,\cdots,T]$ and $\epsilon_{t}\sim\mathcal{N}(0,\mathbf{I})$.

By iterative computation, we can directly obtain $y_{t}$. This process can be further marginalized as:

where $\bar{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{i}$.

In the reverse process, we map from $y_{t}$ to $y_{t-1}$ until the segmented image is acquired step by step. The mathematics is defined as follows:

We train a denoising UNet model to predict ${\epsilon}_{\theta}\left({y}_{t},t,{x}_{o}\right)$:

We follow the improved DDPM [39] to simplify Eq. (4)-(7) to define the hybrid objective $L_{\mathrm{hybrid}}=L_{\mathrm{simple}}+L_{\mathrm{vlb}}$. $L_{\mathrm{vlb}}$ learns the term $\Sigma_{\theta}\left(y_{t},t,x_{o}\right)$. Furthermore, inspired by [54], we use FF and a convolution layer to provide an initial static mask $y_{m}$ to reduce the diffusion variance, and its mean square loss is defined as $L_{\mathrm{static}}$. Total loss function $L_{total}$ is defined as follows:

Algorithm 1 provides the training procedure for diffCOD.

In the inference stage, we step-by-step apply Eq. (14) to sample a pure Gaussian noise $y_{t}\sim\mathcal{N}(0,I)$. In addition, we add conditional information related to the image features to guide the inference process. After performing $T$ iterations, we can obtain the segmentation image of the camouflaged object. Using the setting of [39] for the sampling, the inference process of diffCOD is shown in Algorithm 2.

Following the standard practice of COD tasks, we use 3,040 images from COD10K and 1,000 images from CAMO as the training set and the remaining data as the test set.

Evaluation metrics. According to the standard evaluation protocol of COD, we employ the five common metrics to evaluate our model, i.e., structure-measure ($S_{\alpha}$), weighted F-measure ($F_{\beta}^{\omega}$), mean F-measure ($F_{m}$), mean E-measure ($E_{m}$) and mean absolute error ($MAE$). The purpose of structure-measure ($S_{\alpha}$) is to evaluate the structural information of the result and ground truth, including object perception and region perception. Weighted F-measure $F_{\beta}^{\omega}$ is the weighted information of the mean F-measure ($F_{m}$) metric, and these two metrics are a combined assessment of the accuracy and recall of the result. Mean E-measure ($E_{m}$) is able to perform both pixel-level matching and image-level statistics, and is used to calculate the overall and local accuracy of the segmentation results. The mean absolute error ($MAE$) metric is often used to evaluate the average pixel-level relative error between the result and ground truth.

Implementation details. The proposed method is implemented with the PyTorch toolbox. We set the time step as $T$ = 1000 with a linear noise schedule for all the experiments. We use Adam as our model optimizer with a learning rate of 1e-4. The batch size is set to 64. During the training, the input images are resized to 256$\times$256 via bilinear interpolation and augmented by random flipping, cropping, and color jittering.

Baselines. Our diffCOD is compared with 11 recent state-of-the-art methods, including CPD [56], EGNet [62], SINet [13], MINet [41], PraNet [15], PFNet [37], LSR [36], ERRNet [25], NCHIT [60], CubeNet [66], CRNet [18]. For a fair comparison, all results are either provided by the authors or reproduced by an open-source model re-trained on the same training set with the recommended setting.

The quantitative comparison of our proposed diffCOD with 11 state-of-the-art methods is shown in Table 1. Our method achieves superior performance over other competitors, indicating that our model can generate high-quality camouflaged segmentation masks compared to previous methods. For the largest COD10K dataset, our method shows a substantial performance jump, with an average increase of 4.8%, 12.8%, 9.5%, 6.4% and 19.1% for $S_{\alpha}$, $F_{\beta}^{\omega}$, $F_{m}$, $E_{m}$ and $MAE$, respectively. For another recent large-scale NC4K dataset, diffCOD also outperforms all methods, increasing by 3.4%, 7.1%, 6.1%, 4.0% and 14.8% on average for $S_{\alpha}$, $F_{\beta}^{\omega}$, $F_{m}$, $E_{m}$ and $MAE$, respectively. In addition, the most significant increases in the CAMO dataset were seen in the $F_{\beta}^{\omega}$ and $MAE$, with improvements of 10.2% and 11.3%, respectively. CHAMELEON is the smallest COD dataset, therefore most of the methods perform inconsistently on this dataset, our method increases 3.0%, 6.2%, 4.0%, 2.6% and 21.2% for $S_{\alpha}$, $F_{\beta}^{\omega}$, $F_{m}$, $E_{m}$ and $MAE$, respectively.

Figure 4 shows a comprehensive visual comparison with current state-of-the-art methods. It can be found that our method achieves competitive visual performance in different types of challenging scenarios. Our diffCOD is able to guarantee the integrity and correctness of recognition even under difficult conditions, such as single object (e.g., row 1-4), multi-objects (e.g., row 5-8), small object (e.g., row 9-11). Nature’s camouflaged organisms often have strange traits, such as tentacles, tiny spikes, etc. Past models have blurred the recognition of edge parts even if the location of the target is correctly targeted. However, we are surprised by the advantages of diffCOD in terms of detailed textures. As shown in Figure 3, our method is able to accurately identify every subtlety, and it can depict the textures of the object in extremely fine detail, solving the blurring problem of segmentation masks in other methods.