MirrorNet: Bio-Inspired Camouflaged Object Segmentation

In biological vision studies, there exist viewpoint-invariant theories and viewpoint-dependent theories [37, 38, 39]. In viewpoint-invariant theories, once a particular object has been stored, the recognition of that object from novel viewpoints should be unaffected, provided that the necessary features can be recovered from that view. In view-dependent theories, once a particular object has been stored, recognition of that object from novel views may be impaired, relative to recognition of previously stored views. However, these theories were formed from the simple setting with the outstanding stimuli (non-camouflaged objects) and the clear background, i.e., white background. Indeed, it is natural for human vision to easily detect non-camouflaged objects such as salient objects since they are outstanding from the background, while it is harder for human to detect camouflaged objects since they are similar to the background. In other words, the visual difference between the background and the non-camouflaged object(s) should be larger than the one between the background and the camouflaged object(s). However, when we flip the images, humans are not fooled by the natural settings and notice differences in the images. Therefore, the flipped images accidentally break the natural layout which leads to the larger distance between the background and the camouflaged objects.

To prove this conclusion intuitively, we compute the visual difference between the main objects (the camouflaged and the non-camouflaged objects) with the background. In particular, we adopt the well-known deep learning model FCN [40] and its variants to compute the difference distance. We use the pretrained model on PASCAL-VOC [41] (with 20 semantic classes and the background class). We extract the confidence score at the second last layer of FCN ($h\times w\times 21$). Then, we compute the mean semantic vectors $s_{bg}$, $s_{fg}$ for the background and the camouflaged/non-camouflaged object regions (obtained from the ground truth maps), respectively. This ends up with 21-dim vector for both regions. $\ell_{2}$ normalization is applied for both $s_{bg}$ and $s_{fg}$. Then the visual difference $d$ between the main objects (the camouflaged/non-camouflaged objects) with the background is computed as: $d=\|s_{fg}-s_{bg}\|_{2}$, where $\|.\|_{2}$ is the Euclidean distance. We also compute the difference between the camo/non-camo object with the background in terms of color (RGB and $l\alpha\beta$), and texture (texton [42]).

Table I shows the visual difference with different settings. The difference in terms of color is much smaller than the ones of features extracted from deep learning models. $l\alpha\beta$ shows a larger distance than RGB in the color space. Meanwhile, the textural features, i.e., Texton [42], yield the larger distance which indicates the usefulness of texture in the task of camouflaged object segmentation. The even larger distances in feature space show that features from deep learning models may be helpful to detect and segment camouflaged objects. Note that the features extracted from FCN, a CNN-based model, actually capture the information of textures and edges. In fact, FCN and CNN networks contain convolutional layers which learn spatial hierarchies of patterns by preserving spatial relationships. In these networks, the first convolutional layer can learn basic elements such as texture and edges [44]. Then, the second convolutional layer can learn patterns composed of basic elements learned in the previous layer. The training process continues until the deep network learns significantly complex patterns and abstract visual concepts. This demonstrates the importance of low-level information such as texture and edges in the task of camouflaged object segmentation.

In addition, the difference values of camo and camo-flip with the background are identical in terms of color. It shows that the flipped stream is not useful in terms of color. However, when we look at the difference in the feature space, the camo and non-camo values are different (non-camo value is slightly better). It shows that the camo flip in the feature space may be helpful for the task.

As a closer look, the distance between main object and background is larger in non-camouflaged and smaller in camouflaged images in all FCN and CRF-RNN settings. In camouflaged images, most pixels of camouflaged objects are classified as background class which leads to the small distance. Meanwhile, most pixels of the main non-camouflaged objects are classified with non-background classes which cause the larger distance. We are also interested in the horizontally flipped images. Therefore, we also compute the visual difference between the background and the camouflaged objects in the horizontally flipped images. As shown in Table I, the camouflaged-flipped images generally produce larger distance. It seems that the horizontally flipped images may contain useful information for the segmentation task. Note that the images of camouflaged objects are taken with the natural settings. Therefore, the flipped images accidentally break the natural layout which lead to the larger distance between the background and the camouflaged objects. This one absolutely paves way to our proposed method.

The camouflaged object contains intrinsic and extrinsic information. The intrinsic information means the difference between the camouflaged object and the background. The small distance may be observed in the color space. The distance may be larger in another space, i.e., deep learned feature space, semantic space. Meanwhile, the extrinsic information means the external change onto the camouflaged object, for example, rotating, translating, and flipping, so that the camouflaged object can be better recognized. Therefore, our proposed framework captures both intrinsic and extrinsic information of the camouflaged object.

Figure 3 depicts the overview of our proposed MirrorNet, the bio-inspired network for camouflaged object segmentation. The MirrorNet actually consists of two streams, the main stream for original image segmentation and the mirror stream for flipped image segmentation, whereas the horizontally flipped image stream exploits the bio-inspired effort to break the natural setting of camouflaged objects. Each stream traverses through the camouflaged object proposal, camouflaged object segmentation and yields the segmentation masks. The two output masks from the two streams are finally fused to produce a pixel-wisely accurate camouflage map which uniformly covers camouflaged objects. Here, the camouflaged object segmentation targets at the intrinsic information. Meanwhile, the mirror stream aims to capture the extrinsic information.

Since non-camouflaged object segmentation attracts most attention compared to camouflaged object segmentation, there are only a few relevant datasets, and most of them have the problem of too few samples. Therefore, we adopt the recently built CAMO dataset [2] which proposed and benchmarked by the previous state-of-the-art camouflaged object segmentation, for the training of instance segmentation framework. The dataset is divided into camouflage and non-camouflage categories, each containing 1,000 training and 250 test set, and a total of 2,500 manually annotated ground truths. Most of the dataset images are mammals, insects, birds, and aquatic animals, each with approximately similar proportions, and a small number of reptiles, human art, soldiers, and amphibians. The diversity of species in this dataset makes our model very adaptable, but it must be pointed out that it also has insufficient samples compared to mainstream datasets like COCO [4].

We first compute the number of connected components on the binary ground truth maps of CAMO dataset (the training part). There are many scenarios where multiple connected components belong to one instance. To avoid cherry pick the training samples, we exclude all training images with more than 2 connected components. Then, we compute the bounding box for the single component in the images. In order to increase the number of training samples, we introduce the Augmentation in Wild. In particular, we first perform translating and flipping instances. Furthermore, it is essentially different from data augmentation for non-camouflaged objects, because to determinate whether an object is camouflaged is not only depends on its own features, but also its surroundings. Therefore, we also clone the object instances and place them onto different image regions with a small color difference in the background. Figure 4 shows some samples of augmented data. In this way, we increase the number of training samples, thus alleviating the problem of insufficient data.

To demonstrate the generality and flexibility of our proposed MirrorNet, we employ various recent state-of-the-art backbones, i.e., ResNet [10] and ResNeXt [5]. Our implementation is based on the published code of Mask R-CNN ³³3https://github.com/facebookresearch/maskrcnn-benchmark, which can be adopted to train camouflaged object proposal and camouflaged object segmentation components. Furthermore, we replace the ROI Align layer with a Precise RoI Pooling layer (PrRoI) [45].

The training process is conducted by fine-tuning the available pre-trained model on camouflage images and non-camouflage images of the CAMO dataset [2] and our Data Augmentation in the Wild. In particular, we set the size of each mini-batch to 256 and used the Stochastic Gradient Descent (SGD) optimization with a moment $\beta=0.9$ and a weight decay of $0.0001$. We trained our network for 120k iterations with the base learning rate of $0.00125$, which is decreased by $10$ times every time we reach the next steps at 80k iterations and 100k iterations. We also remark that we implemented our method in PyTorch, and conducted all the experiments on a computer with a 2.40GHz processor (Intel Xeon CPU E5-2620), 64 GB of RAM, and one GeForce GTX TITAN X GPU.

We used the entire the testing images in the CAMO dataset for the evaluation. We note that the zero-mask ground-truth labels (all pixels have zero values) are for the non-camouflaged object images.

Similarly to [2], we used the F-measure ($F_{\beta}$) [46], Intersection Over Union (IOU) [40], and Mean Absolute Error (MAE) as the metrics to evaluate obtained results. The first metric, F-measure, is a balanced measurement between precision and recall as follows:

Note that we set $\beta^{2}=0.3$ as used in [46, 22] to put an emphasis on precision. IOU is the area ratio of the overlapping against the union between the predicted camouflage map and the ground-truth map. Meanwhile, MAE is the average of the pixel-wise absolute differences between the predicted camouflage map and the ground-truth.

For MAE, we used the raw grayscale camouflage map. For the other metrics, we binarized the results depending on two contexts. In the first context, we assume that camouflaged objects are always present in every image like salient objects; we used an adaptive threshold [47] $\theta=\mu+\eta$ where $\mu$ and $\eta$ are the mean value and the standard deviation of the map, respectively. In the second context which is much closer to a real-world scenario, we assume that the existence of camouflaged objects is not guaranteed in each image; we used the fixed threshold $\theta=0.5$.

As proposed in [35], we also use E-measure ($E_{\phi}$) [48], S-measure ($S_{\alpha}$) [49], and weighted F-measure ($F_{\beta}^{\omega}$) [50] as alternative metrics.

We conduct the experiments on CAMO dataset with two settings, namely, only camouflaged object set, and the full set. To investigate the impact of different backbones on our proposed MirrorNet, we fine-tuned pre-trained models of ResNet (ResNet-50, ResNet-101) [10], and ResNeXt (ResNeXt-101, ResNeXt-152) [5] on CAMO respectively. It is interesting for the computer vision community to see the performance of different frameworks in the camouflaged object segmentation problem. Particularly, we compare our MirrorNet with ANet family networks [2] (denoted by ANet-DHS, ANet-DSS, ANet-SRM, ANet-WSS), the-state-of-the-art for camouflaged object segmentation, and the original FCNs (DHS [51], DSS [52], SRM [16], and WSS [53]). All these methods were also fine-tuned on the CAMO dataset with their published parameters.

As shown in Table II, ANet variants outperform FCN-finetuned methods and ANet variants with the same backbone. Meanwhile, our proposed MirrorNet significantly outperforms all baselines. MirrorNet achieves the best result with ResNeXt-152 backbone. This is totally consistent with our discussion in Section III-A, i.e., the better backbones tend to work better in MirrorNet. Note that all baselines, i.e., FCN-finetuned or ANet variants are trained and tested separately on different sets of data, for example, training and testing on only camouflaged image sets, or training and testing on the full set. Meanwhile, our MirrorNet and its variants are only trained on the full set. However, their performance is also significantly better in the only camouflaged image set.

In this subsection, we investigate the impact of different components in our proposed MirrorNet such as dual stream, data augmentation, and RoI pooling mechanisms. Experimental results are shown in Table III. We remark that we use the ResNeXt-152 backbone for all compared methods.

Dual Stream: We investigate the performance of each stream in the MirrorNet, namely, main stream and mirror stream. We compare our completed MirrorNet, which have both streams, with using single streams (denoted by Baseline 1 for only the main stream and Baseline 2 for only the mirror stream). As shown in Table III, our completed MirrorNet outperforms baselines using individual stream. In addition, the mirror stream output is better than the main stream output alone.

Data Augmentation in the Wild: We suspected that one of the major factors affecting the camouflaged objects segmentation was the limited number of training data. To evaluate the impact of Data Augmentation in the Wild, we compare MirrorNet without using Data Augmentation in the Wild in the training process, denoted by Baseline 3. Table III shows that using Data Augmentation in the Wild in the training process outperforms the one without using the augmented data.

ROI Pooling: We further investigate the performance of different ROI pooling mechanism, namely, RoI Align (denoted by Baseline 4) and PrROI. As also seen in Table III, PrRoI-based MirrorNet surpasses RoI Align-based MirrorNet. In addition, RoI Align-based MirrorNet outperforms the main stream and the mirror stream (with PrRoI). This clearly shows the importance of the mask fusion.

We further compare the performance of MirrorNet with state-of-the-arts reported in [35]. Table IV shows the performance of different methods in terms of E-measure ($E_{\phi}$) [48], S-measure ($S_{\alpha}$) [49], weighted F-measure ($F_{\beta}^{\omega}$) [50], and MAE [2]. Note that these metrics were proposed in [35]. As can be seen, the recently proposed methods tend to achieve better results. Our proposed method, MirrorNet, achieves the best performance in terms of $E_{\phi}$, $S_{\alpha}$, $F_{\beta}^{\omega}$, and MAE. MirrorNet with ResNeXt-152 backbone surpasses the state-of-the-art methods by a remarkable margin in all metrics.

Figure 5 shows the visual comparison of different methods. As illustrated in the figure, our MirrorNet variants yield better results than all state-of-the-art methods. Our results are close to the ground truth and focus on the camouflaged objects. From the results, our MirrorNet successfully segments the camouflaged objects images in CAMO dataset [2]. As discussed in [2], CAMO dataset consists of images from challenging scenarios such as object appearance (the camouflaged object has similar color and texture with the background), background clutter, shape complexity, small object, object occlusion, multiple objects, and distraction. This demonstrates that our MirrorNet can well handle a variety of camouflaged objects.