Frozen CLIP: A Strong Backbone for Weakly Supervised Semantic Segmentation

Weakly supervised semantic segmentation with image-level supervision [51, 1, 8, 55] attracts more attention than other weak supervisions [27, 43] due to less human effort. There are two main solutions: multi-stage approaches [2, 19, 23, 56, 25] and single-stage approaches [55, 39].

The key to the multi-stage solution is to generate high-quality pseudo labels. For example, RIB [23] designed a margin loss in the classification network to reduce the information bottleneck, producing better pixel-level responses from image-level supervision. Du et.al.  [14] proposed a pixel-to-prototype contrast strategy to impose feature semantic consistency to generate higher-quality pseudo labels. MCTformer [52] designed multi-class tokens in the transformer architecture to produce class-specific attention responses to generate refined CAM. Some recent multi-stage approaches attempted to introduce CLIP for this task. CLIMS [51] utilized the CLIP model to activate more complete object regions and suppress highly related background regions. CLIP-ES [29] proposed to use the softmax function in CLIP to compute the GradCAM [41]. With carefully designed text prompts, the GradCAM of CLIP provided reliable pseudo labels to train the segmentation model.

Previous single-stage solutions adopted the ImageNet [11] pre-train model as the backbone to concurrently learn the classification and segmentation tasks, and most of them focused on improving segmentation by providing more accurate supervision or constraining its learning. For example, RRM [55] proposed to select reliable pixels as supervision for the segmentation branch. 1Stage [3] designed a local consistency refinement module to directly generate semantic masks from image-level labels. AA&AR [61] proposed an adaptive affinity loss to enhance semantic propagation in the segmentation branch. AFA [39] designed an affinity branch to refine CAMs to generate better online pseudo labels. ToCo [40] proposed token contrast learning to mitigate over-smoothing in online CAM generation, thus providing better supervision for segmentation.

The CLIP model shows great effectiveness in the multi-stage solution, but using it as a single-stage solution, i.e., directly learning to segment objects with image-level supervision, is not explored.

Fully supervised semantic segmentation aims to segment objects using pixel-level labels as supervision. Most previous approaches are based on Fully Convolutional Network (FCN) [31] architecture, such as DeepLab [6], PSPNet [62] and UperNet [50]. Recent approaches introduced vision transformer [12] as the backbone to improve performance by building global relationships. For example, PVT [46] used a pyramid vision transformer for semantic segmentation. Swin [30] designed a window-based attention mechanism in the vision transformer to effectively improve attention computing. They added a UperNet head [50] for semantic segmentation. MaskFormer [9] and Mask2Former [10] proposed universal image segmentation architecture by combining the transformer decoder and pixel decoder. No matter whether fully or weakly supervised semantic segmentation, almost all segmentation models rely on the ImageNet [11] Pre-train models, and all the model parameters require to train or finetune, which requires a large number of computing costs, while we used a frozen CLIP model as the backbone, leading to much less resource on the computation.

Fig. 2 shows the whole framework of our approach, including four main modules: a frozen CLIP backbone (image encoder and text encoder) to encode the image and text, a classification process to produce initial CAM, a decoder to generate segmentation predictions, a RFM to refine initial CAM to provide pseudo labels for training.

The training pipeline is divided into the following steps:

1. 1.

   First of all, the image is input to the CLIP image encoder for image features. Besides, the foreground and background class labels are used to build text prompts and then input to the CLIP text encoder to generate the corresponding text features. Note here both image and text encoders are frozen during training.

2. 2.

   Then, the classification scores are generated by computing distances between image features (after pooling) and text features. Based on classification scores, GradCAM [41] is utilized to generate the initial CAM.

3. 3.

   Besides, image features from the last layer of each transformer block in the frozen CLIP image encoder are input to our proposed decoder for the final segmentation predictions.

4. 4.

   Simultaneously, the intermediate feature maps from our decoder are used to generate an affinity map. Then, the affinity map is input to our proposed RFM with the multi-head attention maps from each block of the frozen CLIP image encoder.

5. 5.

   Finally, RFM outputs a refining map to refine the initial CAM. After post-processing, the final converted pseudo label from refined CAM is used to supervise the training.

We use the frozen CLIP encoder with ViT-B as the backbone, which is not optimized during training. Therefore, how to design a decoder that interprets CLIP features to semantic features becomes a core challenge. We propose a light decoder based on the transformer architecture to conduct semantic segmentation using CLIP features as input.

Specifically, suppose the input image is $I\in\mathbb{R}^{3\times H\times W}$, $H$ and $W$ represent the height and width of the image, respectively. After passing the CLIP image encoder, we generate the initial feature maps $\left\{F_{\text{init}}^{l}\right\}_{l=1}^{N}$ from the output of each transformer block in the encoder, where $l$ represents the index of the block. Then, for each feature map $F_{\text{init}}^{l}$, an individual MLP module is used to generate new corresponding feature maps $F_{\text{new}}^{l}$:

$$F_{\text{new}}^{l}=W_{\text{fc}}^{1}(\text{ReLU}(W_{\text{fc}}^{2}(F_{\text{init}}^{l}))),$$

(1)

where $W_{\text{fc}}^{1}$ and $W_{\text{fc}}^{2}$ are two different fully-connected layers. $\text{ReLU}(\cdot)$ is the ReLU activation function.

After that, all new feature maps $\left\{F_{\text{new}}^{l}\right\}_{l=1}^{N}$ are concatenated together, which are then processed by a convolution layer to generate a fused feature map $F_{u}$:

$$F_{u}=\text{Conv}(\text{Concat}[F_{\text{new}}^{1},F_{\text{new}}^{2},..,F_{\text{new}}^{N}]),$$

(2)

where $F_{u}\in\mathbb{R}^{d\times h\times w}$, where $d$, $h$, and $w$ represent the channel dimension, height, and width of the feature map. $\text{Conv}(\cdot)$ is a convolutional layer, $\text{Concat}[\cdot]$ is the concatenation operation.

Finally, we design several sequential multi-head transformer layers to generate the final prediction $P$:

$$P=\text{Conv}(\phi(F_{u}))\uparrow,$$

(3)

where $P\in\mathbb{R}^{C\times H\times W}$, $C$ is the class number including background. $\phi$ represents the sequential multi-head transformer blocks [12], each block contains a multi-head self-attention module, a feed-forward network, and two normalization layers, as shown in the upper right corner of Fig. 2. $\uparrow$ is an upsample operation to align the prediction map size with the original image.

To provide supervision for the prediction $P$ in Eq. (3), we generate the pixel-level pseudo label from the initial CAM of the frozen backbone. The frozen backbone can only provide static CAM, which means pseudo labels used as supervision cannot be improved during training. The same errors in pseudo labels lead to uncorrectable optimization in the wrong directions. Therefore, we design the Frozen CLIP CAM Refinement module (RFM) to dynamically update CAM to improve the quality of pseudo labels.

We first follow [29] to generate the initial CAM. For the given Image $I$ with its class labels, $I$ is input to the CLIP image encoder. The class labels are used to build text prompts and input to the CLIP text encoder. Then, the extracted image features (after pooling) and text features are used to compute the distance and further activated by the softmax function to get the classification scores. After that, we use GradCAM [41] to generate the initial CAM $M_{\text{init}}\in\mathbb{R}^{(|C_{I}|+1)\times h\times w}$, where $(|C_{I}|+1)$ indicates all class labels in the image $I$ including the background. More details can be found in our supplementary material or [29].

To thoroughly utilize the prior knowledge of CLIP, the CLIP model is fixed. Although we find that such a frozen backbone can provide strong semantic features for the initial CAM with only image-level labels, as illustrated in Fig. 3(a), $M_{\text{init}}$ cannot be optimized as it is generated from the frozen backbone, limiting the quality of pseudo labels. Therefore, how to rectify $M_{\text{init}}$ during training becomes a key issue. Our intuition is to use feature relationships to rectify the initial CAM. However, we cannot directly use the attention maps from the CLIP image encoder as the feature relationship, as such attention maps are also fixed. Nevertheless, the decoder is constantly being optimized, and we attempt to use its features to establish feature relationships to guide the selection of attention values from the CLIP image encoder, keeping useful prior CLIP knowledge and removing noisy relationships. With more reliable feature relationships, the CAM quality can be dynamically enhanced.

In detail, we first generate an affinity map based on the feature map $F_{u}$ in Eq. (2) from our decoder:

$$A_{f}=\text{Sigmoid}(F_{u}^{T}F_{u}),$$

(4)

where $F_{u}\in\mathbb{R}^{d\times h\times w}$ is first flattened to $\mathbb{R}^{d\times hw}$. $\text{Sigmoid}(\cdot)$ is the sigmoid function to guarantee the range of the output is from 0 to 1. $A_{f}\in\mathbb{R}^{hw\times hw}$ is the generated affinity map. $T$ means matrix transpose.

Then we extract all the multi-head attention maps from the frozen CLIP image encoder, denoted as $\left\{A_{s}^{l}\right\}_{l=1}^{N}$ and each $A_{s}^{l}\in\mathbb{R}^{hw\times hw}$. For each $A_{s}^{l}$, we use $A_{f}$ as a standard map to evaluate its quality:

$$S^{l}=\sum\limits_{i=1}^{hw}\sum\limits_{j=1}^{hw}\left|A_{f}(i,j)-A_{s}^{l}(i,j)\right|,$$

(5)

We use the above $S^{l}$ to compute a filter for each attention map:

$$G^{l}=\begin{cases}1,&\text{ if }S^{l}<\frac{1}{N-N_{0}+1}\sum\limits_{l=N_{0}}^{N}S^{l}\\ 0,&\text{ else }\end{cases},$$

(6)

where $G^{l}\in\mathbb{R}^{1\times 1}$, and it is expanded to $G_{e}^{l}\in\mathbb{R}^{hw\times hw}$ for further computation. We use the average value of all $S^{l}$ as the threshold. If the current $S^{l}$ is less than the threshold, it is more reliable, and we set its filter value as $1$. Otherwise, we set the filter value as $0$. Based on this rule, we keep high-quality attention maps and remove weak attention maps.

We then combine $A_{f}$ and the above operation to build the refining map:

$$R=\frac{A_{f}}{N_{m}}\sum\limits_{l=1}^{N}G_{e}^{l}A^{l}_{s},$$

(7)

where $N_{m}$ is the number of valid $A_{s}^{l}$, i.e., $N_{m}=\sum_{l=N_{0}}^{N}G^{l}$.

Then, following the previous approaches [29], we generate the refined CAM:

$$M_{f}^{c}=\left(\frac{R_{\text{nor}}+R^{T}_{\text{nor}}}{2}\right)^{\alpha}\cdot M_{\text{init}}^{c},$$

(8)

where $c$ is the specific class, $M_{f}^{c}$ is the refined CAM for class $c$, $R_{\text{nor}}$ is obtained from $R$ using row and column normalization (Sinkhorn normalization [42]). $\alpha$ is a hyper-parameter. This part passes a box mask indicator [29] to restrict the refining region. $M_{\text{init}}^{c}$ is the CAM for class $c$ after reshaping to $\mathbb{R}^{hw\times 1}$. Finally, $M_{f}$ is input to the online post-processing module, i.e., pixel adaptive refinement module proposed in [39], to generate final online pseudo labels $M_{p}\in\mathbb{R}^{h\times w}$.

In this way, our RFM uses the updated feature relationship in our decoder to assess the feature relationship in the frozen backbone to select reliable relationships. Then, higher-quality CAM can be generated with the help of more reliable feature relationships for each image. Fig. 3 shows the detailed comparison of generated CAM using different refinement methods. Our method generates more accurate responses than the static refinement method proposed in [29] and the initial CAM.

In our RFM, we use the affinity map $A_{f}$ to select the attention map and build the final refining map. Therefore, the effectiveness of $A_{f}$ directly determines the quality of the online pseudo labels. Considering $A_{f}$ is generated using the feature map $F_{u}$ in our decoder, and is a learnable module, we propose a learning process for $A_{f}$ that uses the converted online pseudo label from $M_{p}$ as supervision.

Specifically, $M_{p}$ is first converted to the pixel-wise affinity label for each pair of pixels:

$$\hat{A}=O_{h}(M_{p})^{T}O_{h}(M_{p}),$$

(9)

where $O_{h}(\cdot)$ is one-hot encoding and $O_{h}(M_{p})\in\mathbb{R}^{C\times hw}$, $\hat{A}\in\mathbb{R}^{hw\times hw}$ is the affinity label. $\hat{A}(i,j)=1$ means pixel $i$ and $j$ has the same label, otherwise, $\hat{A}(i,j)=0$.

Based on the above label $\hat{A}$ and the online label $M_{p}$, the whole loss function of our WeCLIP is:

$$\mathcal{L}=\mathcal{L}_{ce}(P,M_{p}\uparrow)+\lambda\mathcal{L}_{ce}(A_{f},\hat{A}),$$

(10)

where $\mathcal{L}_{ce}$ is the cross-entropy loss, $M_{p}\uparrow\in\mathbb{R}^{H\times W}$, and $\lambda$ is the weighting parameter. $P$ is the prediction in Eq. (3). With Eq. (10), more accurate feature relationships are established for higher-quality pseudo labels. In turn, with better pseudo labels, more precise feature relationships are established. Thus, our decoder and RFM can benefit from each other to boost the training.

Following the setting in most previous weakly supervised semantic segmentation approaches [14, 19, 23], two datasets are used to evaluate our approach: PASCAL VOC 2012 [15] and MS COCO-2014 [28]. PASCAL VOC 2012 is appended with SBD [17] to expand the dataset, and the whole dataset contains 10,582 training images, 1,446 validation images, and 1,456 test images with 20 foreground classes. The MS COCO-2014 dataset includes approximately 82,000 training images and 40,504 validation images with 80 foreground classes.

Mean Intersection-over-Union (mIoU) is applied as the evaluation criterion.

We use the frozen CLIP backbone with the ViT-16-base architecture [13], $N$ is a fixed number that equals $12$. For training on the PASCAL VOC 2012 dataset, the batchsize is set as $4$, and the maximum iteration is set as $30,000$. For training on the MS COCO-2014 dataset, we set batchsize as $8$, and the maximum iteration as $80,000$.

All other settings adopt the same parameters for two datasets during training: We use AdamW [32] as the optimizer, the learning rate is $2e^{-3}$ with weight decay $1e^{-3}$, and all images are cropped to $320\times 320$ during training. $\lambda$ in Eq. (10) is set as $0.1$, The dimension of the MLP module (Eq. (1)) in our decoder is set as $256$. In $\phi$ of Eq. (3), three transformer encoder (the multi-head number is $8$) layers are cascaded to generate the final feature map, and each layer’s output dimension is $256$. $N_{0}$ in Eq. (6) is set as $6$. $\alpha$ is set as $2$ in Eq. (8) following [29].

During inference, we use the multi-scale with $\left\{0.75,1.0\right\}$. Following previous approaches [39, 53, 40], DenseCRF [21] is used as the post-processing method to refine the prediction.

In Tab. 1, we compare our approach with other state-of-the-art approaches on the PASCAL VOC 2012 dataset. It can be seen that our WeCLIP reaches 76.4% and 77.2% mIoU on val and test sets, both of which significantly outperform other single-stage approaches by a large margin. Specifically, compared to ToCo [40], the previous state-of-the-art single-stage approach, our WeCLIP brings 5.3% and 5.0% mIoU increase on val and test set, respectively. Besides, CLIP-ES [29] is the previous state-of-the-art multi-stage approach, and it is also a CLIP-based solution. Our approach performs much better than it, with 3.6% and 3.3% mIoU increase.

Tab. 2 shows the comparisons between our approach and previous state-of-the-art approaches on MS COCO-2014 val set. Our approach achieves new state-of-the-art performance, reaching 47.1% mIoU. Compared to other single-stage approaches, our WeCLIP brings more than 4.8% mIoU increase, which is a significant improvement. More importantly, our WeCLIP also outperforms other multi-stage approaches by a clear margin with fewer training steps. Considering our WeCLIP uses a frozen backbone, it shows great advantages to this task.

In Tab. 3, we compare the training cost between our approach and other state-of-the-art approaches on the PASCAL VOC 2012 dataset. It can be seen that our approach only needs 6.2G GPU memory, while other approaches require at least 12G GPU memory. ToCo [40] has less training time than us, but its GPU memory is much higher than our WeCLIP. More importantly, ToCo [40] spent 4 hours with 20,000 training iterations, while our WeCLIP spent 4.5 hours with 30,000 iterations, which also shows the high training efficiency of our approach.

In Fig. 4, we show some qualitative comparisons between our approach and other approaches on the PASCAL VOC 2012 and MS COCO-2014 val set. The visual results show that our WeCLIP generates more accurate object details than ToCo [40] for both the two datasets.

We conduct ablation studies on the PASCAL VOC 2012 val set to evaluate the effectiveness of our approach. CRF is not used to refine the final prediction.

Tab. 4 shows the influence of our proposed decoder and RFM. As a single-stage approach, the decoder is necessary. We cannot generate the prediction without it. Besides, introducing RFM brings a clear improvement, with a 6.2% mIoU increase. Since RFM is designed to improve the online pseudo labels, this increase also evaluates its effectiveness in generating higher quality pseudo labels.

Tab. 5 reports the influence of the number of transformer layers in our decoder, i.e., $\phi$ in Eq. (3). The performance increases when the layer number increases to 3. This is because the limited size of the decoder cannot capture enough feature information, and it is easy to under-fit the features. With the increase of layer number, the decoder learns better feature representation. However, the performance drops if the layer number is larger than 3. One possible reason is that deeper decoder layers cause the over-fitting problem. Thus, it is reasonable that the performance drops after increasing to 4 or 5 for $\phi$.

In Tab. 6, we evaluate the effectiveness of our refining map. When only $A_{s}$ is used, it means that all attention maps are selected to refine the CAM, i.e., the same process proposed in [29], it generates 71.8% mIoU score. Note that such a process is a static operation, which is not optimized during training. Introducing $G_{e}$ and $A_{f}$ clearly improves it with $0.5\%$ and $2.5\%$ mIoU increase, respectively. Finally, combining $A_{f}$, $G_{e}$, and $A_{s}$ using Eq. (7) generates much better results than others, showing the effectiveness of our refining method. More importantly, using the affinity map from our decoder provides a dynamic refinement strategy, making the refinement process optimized during training.

We also use our WeCLIP to tackle fully-supervised semantic segmentation. For fully-supervised semantic segmentation, it provides accurate pixel-level labels, so we remove the frozen text encoder and our RFM, only keeping the frozen image encoder and our decoder. Besides, the loss function removes the part related to $\hat{A}$. The framework can be found in our supplementary material.

In Tab. 7, we evaluate our approach on PASCAL VOC 2012 set for fully-supervised semantic segmentation. Since our approach utilizes a frozen backbone, it has less trainable parameters, but high-level segmentation performance is maintained, showing its great potential for fully-supervised semantic segmentation.

To illustrate why the vision feature from frozen CLIP can be directly used for semantic segmentation, we show some feature visualization results to compare the difference between the CLIP features and ImageNet features in Fig. 5. We randomly select 200 images from the PASCAL VOC 2012 train set. Without any training or finetune, we use ViT-B as the backbone and directly initialize it with frozen pre-train weights. It can be found that features belonging to the same class, pre-trained by CLIP, are denser and clustered, while features belonging to the same class, pre-trained by ImageNet, are more sparse and decentralized. Fig. 5 indicates that the extracted features from the CLIP model can better represent semantic information for different classes, making features belonging to different classes not confused. With such discriminative features, It is more convenient to conduct segmentation tasks.