Chain of Visual Perception: Harnessing Multimodal Large
Language Models for Zero-shot Camouflaged Object Detection

Prompted by the powerful generalized ability of LLMs (Devlin et al., 2019; Lu et al., 2019; Brown et al., 2020; Zhang et al., 2023a) in NLP, VFMs (Oquab et al., 2023; Kirillov et al., 2023b; Radford et al., 2021; Li et al., 2022) have emerged. The integration of LLMs and VFMs has facilitated the advancement of MLLMs (Huang et al., 2023b; Zhu et al., 2023; Li et al., 2023d; Dai et al., 2023; Zhang et al., 2023c; Zhao et al., 2023; Chen et al., 2023; Liu et al., 2023a; OpenAI, 2023; Li et al., 2023b). MLLMs show case impressive visual understanding through end-to-end training techniques that directly decode visual and text tokens in a unified manner. These foundational models, like GPT, SAM and LLaVA, demonstrated the immense potential of these large-scale, versatile models, trained on extensive datasets to achieve unparalleled adaptability across a wide range of tasks. This paradigm shift, characterized by significant strides in representation learning, has spurred the exploration of task-agnostic models, propelling research into both their adaptation methodologies and the intricacies of their internal mechanisms.

In the field of NLP, to migrate LLMs to downstream tasks without impacting the LLM’s inherent performance, In-context learning (Dong et al., 2022) is a widely used technique. A particularly influential approach within In-context learning is the CoT (Wei et al., 2022; Kojima et al., 2022; Fu et al., 2023). CoT, by designing a series of reasoning steps, guides the LLM to focus on specific content at each step, thereby further stimulating the model’s innate logical reasoning capabilities. Specifically, these works have discovered that by prompting LLMs with designed directives, such as “let’s think step by step”, the reasoning abilities of the LLMs can be further improved and enhanced.

As a field that is more mature than MLLM, LLM has shown that models can be effectively migrated to various downstream tasks in a zero-shot manner, provided that the correct prompting mechanisms are in place. Therefore, to further advance the development of MLLM, our paper explores the upper limits of MLLM performance in visually challenging tasks, specifically COD. Unlike existing methods that use re-training or tuning to migrate MLLMs to downstream tasks (Peng et al., 2023; Pi et al., 2023; Lai et al., 2023), our paper aims to explore how to prompt the MLLMs to stimulate its inherent perception abilities for camouflaged scenes. To this end, we propose CoVP, which first identifies key aspects to consider when inputting language text prompts to enhance MLLM’s understanding of camouflaged scenes. Further, we highlight how to utilize MLLM’s uncertain visual outputs and, from a visual completion standpoint, enhance MLLM’s capability to capture camouflaged objects.

In the past years, there has been significant effort in the COD task (Fan et al., 2020; Mondal, 2020; Zhai et al., 2021; Lv et al., 2021; Yang et al., 2021b; Li et al., 2019; Fan et al., 2022; Zhong et al., 2022; Pang et al., 2022; Bi et al., 2022; Zhang et al., 2022a; Hu et al., 2023; Ji et al., 2023; Zhuge et al., 2023). The technical frameworks for these COD methods can be categorized into two types: CNN-based and Transformer-based approaches. Although the structures of these methods may differ, their core lies in designing advanced network modules or architectures capable of exploring discriminative features. While these methods have achieved impressive performance, the networks lack generality and are task-specific, which limits their generalizability. This means that while they are highly effective for specific tasks, their adaptability to a wide range of different tasks is constrained.

The emergence of a series of foundational models in recent times has signaled to computer vision researchers that it is possible to solve a variety of downstream vision tasks using a single, large-scale model. This trend highlights the potential of leveraging powerful, versatile models that have been trained on extensive datasets, enabling them to handle diverse and complex visual challenges. In line with the trend of technological advancement, this paper explores the generalization capabilities of visual foundational models. We design the MMCPF to generalize the foundational model to the COD task in a zero-shot manner. It’s important to highlight that this paper does not employ methods such as re-training, adapters, or tuning to update the parameters of the visual foundational model for adaptation to the COD task. Instead, we explore how to enhance the perception abilities of the visual foundational model in camouflaged scenes through prompt engineering from both linguistic and visual perspectives, without altering its inherent capabilities.

Existing works like ZSCOD (Li et al., 2023a) and GenSAM (Hu et al., 2024) are also attempting to design a zero-shot COD framework. The pipline of ZSCOD may seem somewhat contrived, potentially limiting its adaptability to real-world scenarios. Specifically, this method artificially divides the COD10K dataset into seen and unseen categories and then trains the network. This undoubtedly leads to the risk of information leakage, thus preventing a true evaluation of whether the network has generalizability. Moreover, its performance to unknown scenarios is very weak, raising concerns among researchers about the robustness of their networks. GenSAM (Hu et al., 2024) also leverages MLLM and VFM to design a zero-shot COD framework. However, a flaw of this framework is that it makes additional modifications to the foundation model (Clip-Surgery (Li et al., 2023c)) used in the framework. Specifically, to enhance the network’s ability to filter noise from camouflaged images and accurately identify foreground pixels, GenSAM introduces an additional attention mechanism to the existing well-established framework. This approach is somewhat similar to an Adapter operation. This could potentially affect the generalizability that the foundation model originally had to some extent. Compared to these two methods, our MMCPF is more flexible in its setup. In MMCPF, we do not make any structural modifications to the MLLM and VFM. Instead, we enhance the MLLM’s perceptual capabilities for camouflaged scenes through the design of the CoVP. This approach ensures the maximal preservation of the MLLM’s inherent generalization ability, and achieve superior performance.

In Fig. 2, our proposed MMCPF is a promptable framework, which is the first framework to successfully generalize the MLLM to the camouflaged scene. Given an image $\mathcal{I}$ containing the camouflaged scene, prompting the MMCPF with the text $\mathcal{T}$, such as “Please find a camouflaged object in this image and provide me with its exact location coordinates”, the MMCPF would locate the camouflaged object at position $\mathcal{P}_{I}$ and generate the corresponding mask $\mathcal{M}$.

To achieve the above purpose, we state that MMCPF needs two foundational models. The first is the MLLM, which can accept the user instruction and output the corresponding result, such as the coordinate information of the target object. To further quantitatively evaluate the performance of MLLM, the second foundational model is the promptable VFM, which can accept the output from MLLM as the prompt, and generate the final mask $\mathcal{M}$. Note that, in MMCPF, both MLLM and VFM are frozen.

In MMCPF, only having MLLM and VFM, despite their powerful capabilities, may still be insufficient to handle the COD task effectively. Specifically, in Fig. 2, using just vanilla text prompts to query the MLLM might yield meaningless results, contributing nothing to the accurate location of camouflaged objects. Additionally, in Fig. 3, the positional coordinates output by the MLLM may carry a degree of uncertainty, encompassing only a part of the camouflaged object. Specifically, for the localization of camouflaged objects, MLLM typically outputs the coordinates of the top-left and bottom-right corners. We have observed that these coordinates do not always fall within the interior of the camouflaged object and their central point only lands on the edge of the camouflaged object sometimes. Consequently, if these coordinates are directly used as point prompts for the VFM, the resulting mask might be incomplete or fragmented. Therefore, to address the above problems, we propose CoVP, which enhance MLLM’s perception of camouflaged scenes from both linguistic and visual perspectives.

Images in camouflaged scenes obviously present visual challenges, making it difficult for MLLMs to detect camouflaged objects. The challenges primarily encompass two aspects. Firstly, for MLLM, we stimulate its understanding of visual content in an image through language. However, designing language prompts suitable for camouflaged scenes remains an area to be explored. The existing work (Cao et al., 2023) attempts to enhance the visual perception capabilities of MLLM by providing semantic information about an image, which contradicts the definition of the COD task, and thus cannot be directly applied. Therefore, our first task is to design how language can be used to enhance the visual perception ability of MLLM.

Secondly, prompting MLLM to visually perceive an image through language represents a challenging cross-modal task, especially when we attempt to generalize MLLM to visually the challenging COD scene. As visualized in Fig. 3, it’s difficult to completely ensure the accuracy of MLLM’s output. Consequently, we design a visual completion (VC) to further enhance the localization capability of MLLM. Unlike the CoT, which only designs mechanisms at the text input of the LLM to enhance its language reasoning ability, CoVP attempts to improve MLLM’s perception of camouflaged scenes more comprehensively, working at both the input and output and from linguistic and visual perspectives.

We employ three public benchmark COD datasets to evaluate the perceptual capabilities of MMCPF in camouflaged scenes. These datasets include CAMO (Le et al., 2019), COD10K (Fan et al., 2020) and NC4K (Lv et al., 2021). CAMO is a subset of the CAMO-COCO, specifically designed for camouflaged object segmentation. COD10K are collected from various photography websites and are classified into 5 super-classes and 69 sub-classes. NC4K features more complex scenarios and a wider range of camouflaged objects. To further demonstrate the generalization capability of our proposed MMCPF, we extend our validation to other datasets which may contain different camouflaged scene, MoCA-Mask (Cheng et al., 2022) and OVCamo (Pang et al., 2023), to assess our method’s performance. We adopt three widely used metrics to evaluate our method: structure-measure ($S_{\alpha}$) (Fan et al., 2017), weighted F-measure ($F^{w}_{\beta}$), and mean absolute error (MAE). To ensure the reproducibility of our MMCPF, thereby positively impacting the community, we select open-source models, Shikra (Chen et al., 2023) and SAM-HQ (Ke et al., 2023) for both the MLLM and VFM.

Selecting appropriate comparison methods is crucial to demonstrate the contribution of our proposed MMCPF to the community. The core of our MMCPF is to generalize MLLM and VFM to camouflaged scenes in a zero-shot manner. Since the MLLM and VFM we chose are not specifically designed for camouflaged scenes, we first compare our method with the zero-shot COD method ZSCOD (Li et al., 2023a) and GenSAM (Hu et al., 2024). Secondly, as we do not retrain the MLLM and VFM on camouflaged datasets when generalizing them to camouflaged scenes, it is appropriate to compare our approach with unsupervised COD methods. Unfortunately, we could not find any unsupervised methods specifically designed for the COD task, so we opt for a comparison with the weakly-supervised method WSCOD (He et al., 2023). Finally, we also compare our method with three fully-supervised approaches, including NCHIT (Zhang et al., 2022b), ERRNet (Ji et al., 2022), and FSPNet (Huang et al., 2023a). This comparison not only helps researchers understand the performance level of our paper but also further clarifies our contribution to the field. By setting our work against a backdrop of various supervisory approaches, we provide a comprehensive view of where MMCPF stands in the context of current COD methodologies and highlight its potential advantages. In addition to comparisons with current state-of-the-art COD methods, we also benchmark against two VCOD methods, SLTNet (Cheng et al., 2022) and MG (Yang et al., 2021a), which can demonstrate the generalization capability of MMCPF. Finally, we further compare MMCPF with Open-Vocabulary COD method OVCoser (Pang et al., 2023).

Finally, we also validate the effectiveness of our MMCPF in the newly proposed Open-Vocabulary COD scenario, OVCamo. As can be seen from Table. 4, our method surpasses some recent fully supervised methods, which fully demonstrates the effectiveness of our approach. Through this series of experiments, the strong generalization ability of our proposed MMCPF has been fully validated.

In Table. 5, Baseline represents our use of the vanilla text prompt, “Please find a camouflaged object in this image and provide me with its exact location coordinates”, to query MLLM, without incorporating visual completion. The results in the first row indicate that using vanilla text prompt alone is insufficient to enable MLLM to perceive camouflaged scenes effectively. Subsequently, we enhance the text descriptions by including attributes of the camouflaged objects, and the text prompt is “This image may contain a camouflaged object whose shape, color, texture, pattern and movement closely resemble its surroundings, enabling it to blend in. Can you identify it and provide its precise location coordinates?”. The results from the second and third rows demonstrate a further improvement. After that, considering the issue of polysemy in descriptions, we modify the text prompts as “This image may contain a concealed object whose shape, color, texture, pattern and movement closely resemble its surroundings, enabling it to blend in. Can you identify it and provide its precise location coordinates?”. Using these two types of prompts in tandem to cue the MLLM, we observe an additional enhancement in performance. Finally, we generate synonymous prompts based on the first two text types to further cue the MLLM, thereby improving performance. The diverse text prompt is “This image may contain a camouflaged object whose shape, color, pattern, movement and texture bear little difference compared to its surroundings, enabling it to blend in. Please provide its precise location coordinates.”.

In MMCPF, we also implement visual completion to further enhance the MLLM’s ability to perceive camouflaged objects. The results in the sixth row demonstrate that incorporating visual completion can lead to further performance improvements. Fig. 7 visually illustrates the effectiveness of visual completion, showcasing how this component of our approach significantly aids in the accurate detection and delineation of camouflaged objects.