Improving Semi-Supervised Semantic Segmentation with Dual-Level Siamese Structure Network

Following SSS works (Chen et al., 2021; Yang et al., 2022; Liu et al., 2022b), we use both a small fraction of labeled data $\mathcal{D}_{l}=\{(X_{i},\bm{T}_{i})\}_{i=1}^{M}$ and a large fraction of unlabeled data $\mathcal{D}_{u}=\{X_{i}\}_{i=1+M}^{N+M}$ . $X_{i}$ denotes an image, and $\bm{T_{i}}$ represents its ground-truth label if $X_{i}$ is a labeled image. $N$ and $M$ indicate the number of labeled and unlabeled images, respectively, where $N\gg M$ in most cases. To facilitate the calculation of loss functions, we represent each pixel in an image as a vector $\bm{x}$ since a pixel has values in different channels. Thus, in subsequent sections, we represent each pixel as a vector $\bm{x}$ with $\bm{t}$ as its one-hot ground-truth label. Given an image $X=[\bm{x}_{i}]$ with the size of $W\times H$ where $W$ and $H$ are the width and height, we denote the pixel by $\bm{x}_{i},i\in\{1,...,W\times H\}$. The latent high-level feature $\bm{z}$ corresponding to $\bm{x}$ is obtained by an encoder $f(\bm{x}|\theta)$ where $\theta$ is the learnable parameters of the encoder. We yield the predicted logits $\bm{h}$ by feeding the latent representations $\bm{z}$ into a decoder $g(\bm{z}|\varphi)$ where $\varphi$ is the learnable parameters of the decoder. Finally, a softmax layer is added to obtain the ultimate probability for each class, i.e., $\bm{y}={\rm softmax}(\bm{h})$.

Given a labeled image, we use a supervised cross-entropy loss,

where $\mathcal{C}=\{1,\ldots,C\}$ and $C$ is the total number of classes. For a unlabeled image, a simple way to generate their pseudo labels $\hat{\bm{t}}_{i}$ is to apply a one-hot operation to the predictions, i.e., $\bm{y}_{i}$. For the $i$-th pixel of an unlabeled image, we represent the predicted probability of the $i$-th pixel belonging to the $j$-th class as $y_{ij}$ . Specifically, we use the following operation to generate pseudo labels:

where $c$ denotes the maximal probability within the class $j\in\mathcal{C}$, the $\hat{\bm{t}}_{i}=[\hat{t}_{ij}]$ is the one-hot pseudo label.

To fully exploit the potential of available unlabeled data, we propose to use DSSN for extracting pixel-wise contrastive positive pairs in different abstraction levels. The low-level image is subjected to two-view strong augmentations,

where $\bm{x}_{i}^{ls1}$ denotes the strong augmented low-level pixel in the first view. The output, ${\rm AugL}_{s}(\cdot)$, is random. ${\rm AugL}_{s}(\cdot)$ generates varying outputs using the same input to augment the data diversity. This increases the diversity, resulting in an improvement in the robustness and generalization ability of the training model.

We use two-view augmented images to obtain its decoded logits,

Analogous to (Hjelm et al., 2019), we apply the contrastive objective, i.e., $\mathcal{L}_{\rm cl}$ to pairwise pixels for learning better representations:

where $d(\cdot,\cdot)$ is a similarity score of a pair of logits. $\bm{h}_{i}^{ls1}$ and $\bm{h}_{i}^{ls2}$ are belong to positive pairs $(i,i)\in\mathcal{P}$ while $\bm{h}_{i}^{ls1}$ and $\bm{h}_{j}^{ls2}$ are negative pairs $(i,j)\in\mathcal{N},\forall\,i\neq j$. We use $\mathcal{P}$ and $\mathcal{N}$ to denote the sets of positive and negative pairs, respectively.

Inspired by BYOL (Grill et al., 2020), we only use the positive pairs. The similarity $d(\cdot,\cdot)$ of positive logits is defined by a Gaussian function,

The similarity defined by Eq. (9) implies the similarity is 1 if the pairwise logits are the same while it tends to 0 if their distance is far from each other. From a different perspective, the error $\|\bm{h}^{ls1}_{i}-\bm{h}_{i}^{ls2}\bigr{\|}^{2}_{2}$ of two-view logits is governed by the Gaussian distribution due to the central limit theorem (Walker, 1969), so we also obtain Eq. (9).

Substituting Eq. (9) into Eq. (8) obtains the following loss.

where we only use pixel-wise positive pairs.

For the high-level feature contrastive learning, we obtain the high-level latent feature with the encoder,

where ${\rm AugL}_{w}(\cdot)$ is a weak augmentation for the low-level pixel. The high-level feature is subjected to two-view strong augmentations,

We use the two-view augmented features to obtain its decoded logits, $\bm{h}_{i}^{hs1}=g(\bm{z}_{i}^{hs1}|\varphi)$ and $\bm{h}_{i}^{hs2}=g(\bm{z}_{i}^{hs12}|\varphi)$ . Then, we use them to construct the contrastive loss,

To leverage the four predictions generated by a strongly augmented image, we feed the corresponding weakly augmented image into DSSN. Next, we use the prediction of the weak view to generate its pseudo label and supervise the four strong views. Given our dual-level structure, weak-to-strong pseudo supervision is also performed in both levels. Specifically, we use the pseudo labels of the weak view, denoted as $\hat{\bm{t}}$, to supervise the predictions of the strong views, denoted as $\bm{y}$,.

The weak pseudo supervisions are obtained by

where $(\theta^{\prime},\varphi^{\prime})$ of the teacher are updated from the student $(\theta,\varphi)$ by the exponential moving average (EMA)

where $\alpha$ is a momentum parameter, with $\alpha\in[0,1]$.

The pseudo labels $\hat{\bm{t}}^{lw}$ and $\hat{\bm{t}}^{hw}$ of $\bm{y}^{lw}$ and $\bm{y}^{hw}$ are calculated by using Eqs. (2) and (3), respectively.

The output probability of the strong augmented views, $\bm{y}_{i}^{ls1}$, $\bm{y}_{i}^{ls2}$, $\bm{y}_{i}^{hs1}$, and $\bm{y}_{i}^{hs2}$, are attained by the softmax layer.

The weak-to-strong pseudo-supervision loss functions are

where $m_{ij}$ is a class-wise binary mask to select the pixel with high-confidence score and we show how to obtain it in the next section.

As shown in Fig. 1(c), we show the class-aware pseudo-label generation (CPLG). For the $i$-th pixel, it has different probabilities belonging to different classes. $y_{ij}$ denotes the probability of the $i$-th pixel belonging to the $j$-th class. We observe all pixels in the same class, i.e., in the same channel of network output.

First, we find the pixel class-wisely that has the largest probability in the $j$-th class,

Second, we establish a class-wise threshold $\tau_{j}$ by multiplying the maximum probability by $r$%. Pixels exceeding this class-wise threshold are selected. Additionally, we restrict the maximum probability by $\tau_{\rm low}$ and exclude pixels with a low maximum probability since they indicate lower prediction confidence. Thus, the class-wise threshold $\tau_{j}$ is determined by

where the ratio $r$ and the low bound $\tau_{\rm low}$ are parameters.

Third, we select pixels in each class by $\tau_{j}$, i.e., pixels exceeding $\tau_{j}$ are selected:

The generation of the pseudo label is straightforward by using Eqs. (2) and (3). The refined class-aware pseudo labels are attained by multiplying them, i.e., $m_{ij}\hat{t}_{ij}$, as used in Eqs. (18) and (19). Our CPLG strategy considers the learning status and difficulties of different classes by adjusting thresholds for each class. As a result, we select useful pixels with low thresholds for training, which enhances the accuracy of challenging classes.

Fig. 2 illustrates how we combine two distinct learning strategies for the unlabeled images: contrastive learning and weak-to-strong pseudo supervision.

In this section, we present the DSSN algorithm, which is illustrated in Algorithm 1. It takes a small fraction of labeled data and a large fraction of unlabeled data as input to train the model. The supervised loss between the model prediction on labeled data and the ground truth is computed using Eq. (1). Subsequently, the low-level and high-level contrastive learning losses are calculated using Eqs. (10) and (14), respectively. We then compute the weak-to-strong pseudo-supervision loss using Eqs. (18) and (19). The overall loss term is formulated as follows:

where $\gamma_{1}$ and $\gamma_{2}$ are the trade-off weight. Finally, we update the student model and the teacher model by using the error back-propagation algorithm and EMA, respectively.

Datasets. We evaluate the proposed method on two classical semantic segmentation datasets, i.e., PASCAL VOC 2012 (Everingham et al., 2015) and Cityscapes (Cordts et al., 2016). In particular, PASCAL VOC 2012 (Everingham et al., 2015) has 20 classes of objects and 1 class of background. The standard training, validation and test sets consist of 1,464, 1449 and 1,456 images, respectively. Following the previous work (Yang et al., 2022; Chen et al., 2021; Ke et al., 2020), we also use augmented set SBD (Hariharan et al., 2011) (9,118 images) and original training set (1,464 images) as our full training set (10,582 images). The labels from the SBD (Hariharan et al., 2011) dataset are noise-prone and of low quality. Cityscapes (Cordts et al., 2016) has 19 semantic classes and is mostly intended for understanding urban scenes. It consists of 500 validation images, 1,525 test images, and 2,975 training images. All of the images have well-annotated masks. For a fair comparison with the benchmarks, we follow the partition procedure of CPS (Chen et al., 2021). Specifically, the training set is divided into two partitions by randomly sampling 1/2, 1/4, 1/8, and 1/16 of the total set as the labeled samples and the remaining images as the unlabeled for the blended set.

Implementation details. Following the previous benchmarks CPS (Chen et al., 2021), we adopt DeepLab v3+ (Chen et al., 2018) based on ResNet (He et al., 2016) as the segmentation network for a fair comparison. The backbone i.e., ResNet, is initialized with the weights pre-trained on ImageNet (Deng et al., 2009). The segmentation heads are randomly initialized. During training, each mini-batch contains eight labeled and eight unlabeled images. The stochastic gradient descent (SGD) optimizer is used, and the initial learning rates are set to 0.002 and 0.005 for the PASCAL VOC 2012 and Cityscapes, respectively. In accordance with other works (Chen et al., 2021; Ouali et al., 2020), we employ the following polynomial to decrease the learning rate while training: $(1-{{epoch}}/{{epoch_{\max}}})^{0.9}$. The model is trained for 100 epochs on PASCAL VOC 2012 and 240 epochs for Cityscapes. For weak augmentations, we adopt the same operation as ST++(Yang et al., 2022), where the training images are random flipping and resizing (between 0.5 and 2.0 times), followed by a random crop operation to the resolutions of 513 $\times$ 513 and 801 $\times$ 801 for the two datasets, respectively. We employ several strong augmentation, including random color-jitter, grayscale, Gaussian blur, etc. For strong feature augmentation, we apply a random dropout of 50% on features from the encoder. The unsupervised trade-off weights $\gamma_{1}$ and $\gamma_{2}$ are set as 0.01 and 0.25. In CPLG, $r$ is set to 96% and $\tau_{\rm low}$ is 0.92, respectively.

Additionally, we also apply CutMix (Yun et al., 2019) data augmentation to the student model images. The EMA smoothing factor $\alpha$ is set as 0.996. We follow U²PL (Wang et al., 2022), the supervised loss is cross-entropy on PASCAL, and for Cityscapes the cross-entropy loss is replaced by the online hard example mining loss.

Evaluation. We use the mean of Intersection-over-Union(mIoU) for the validation set to evaluate the segmentation performance for both datasets. Following the previous works (Chen et al., 2021; Yang et al., 2022), we employ the sliding evaluation to examine the efficacy of our model on the validation images from Cityscapes with a resolution of 1024×2048.

To demonstrate the superiority of our proposed DSSN  method, we conduct a comparison with the current state-of-the-art methods across various settings. All results are reported on the validation set for both PASCAL VOC and Cityscapes datasets. Additionally, we present the corresponding baseline at the top of the table, representing the results of purely supervised learning trained on the same labeled data. To ensure a fair comparison, all methods employed the DeepLab v3+ architecture.

PASCAL VOC 2012. We report results of our experiments on the PASCAL VOC 2012 validation dataset in Tables 1 and 2, where we evaluate the mean Intersection over Union (mIoU) for different proportions of labeled samples. Additionally, we present the corresponding baseline at the top of the table, representing the results of purely supervised learning trained on the same labeled data.

Table 1 presents results on the classic PASCAL VOC 2012 dataset. It shows our method significantly outperforms current state-of-the-art methods. When employing ResNet-101 as the backbone, DSSN attains a 5.18% performance gain on the 1/16(92) split which surpass the performance obtain by the (1/3)183 data split in the prior study. Even with more labeled data, the performance differences become less evident; however, the proposed method still demonstrates performance improvements of 2.21% with 1/2 fine annotations over the previous SOTAs.

Table 2 illustrates the results on blender PASCAL VOC 2012 Dataset. Our method shows significant improvement on the 1/16, 1/8, 1/4, and 1/2 splits with ResNet-50, compared to the baseline, with improvements of 15.51%, 10.1%, 6.73%, and 3.57%, respectively. Similarly, with ResNet-101, our method achieves improvements of 12.9%, 9.18%, 6.65%, and 3.01% under the same partitions. Especially, our method shows significant improvements when the ratio of labeled data becomes smaller, such as under 1/8 or 1/16 partition protocols. In particular, when the labeled data is extremely limited,e.g., on the 1/16 partitions, our method achieves remarkable increases of 15.51% and 12.9% compared to the baseline with ResNet-50 and ResNet-101 as the backbone networks, respectively. Furthermore, our method demonstrates a considerable improvement over the previous state-of-the-art PS-MT (Liu et al., 2022a), achieving a margin of 3.88% with ResNet-50 as the backbone, and 1.7% under the 1/8 partition protocol.

Cityscapes. In Table 3, we can see that our method consistently outperforms the supervised baseline by a significant margin, achieving improvements of 12.11%, 7.11%, 4.95%, and 2.13% with ResNet-50 under 1/16, 1/8, 1/4, and 1/2 partition protocols, respectively. Similarly, with ResNet-101, our method shows improvements of 10.22%, 5.38%, 3.62%, and 1.58% under 1/16, 1/8, 1/4, and 1/2 partition protocols, respectively. Furthermore, our method outperforms all other state-of-the-art methods across various settings. Specifically, under 1/8, 1/4, and 1/2 partitions, DSSN achieves a 1.55%, 1.13%, and 1.09% improvement over the previous state-of-the-art PS-MT (Liu et al., 2022a) using ResNet-50, and a 1.29%, 1.02%, and 0.49% improvement using ResNet-101, respectively.

We evaluate DSSN using ResNet-50 on a 1/30 data split, which contained only 100 labeled images. As illustrated in Fig. 3, DSSN outperforms the current state-of-the-art significantly. This result indicates that our method effectively utilizes the unlabeled data through contrastive learning and the class-aware pseudo-label selection strategy (CPLG). Besides, although ReCo (Liu et al., 2022b) and U²PL(Wang et al., 2022) try to construct positive and negative pairs to use contrastive learning, the result shows our DSSN outperform them significantly.

Upon comparing performance on classic PASCAL VOC 2012 and blended training set, we observe that the quality of labeled samples is important. For example, DSSN achieves an exceptional performance of 80.61% by utilizing only 732 high-quality labels. However, even with significantly more labels (5291) from the blended dataset, a comparable score of 80.61% cannot be achieved.

In this subsection, we discuss the contribution of each component to our framework using ResNet-101 and a 1/8 labeled ratio on PASCAL VOC 2012 dataset.

Effectiveness of the DSSN components. We conduct a step-by-step ablation study of each component to comprehensively assess their effectiveness. Table 4 presents the results of our study. Without our proposed dual-Level contrastive learning and CPLG, applying a plain consistency method yields an accuracy of 76.12%. However, employing dual-level contrastive learning leads to an accuracy of 78.33%, while the proposed CPLG results in 78.70%. Combining both dual-level contrastive learning and CPLG produces the highest accuracy of 79.58%, demonstrating the effectiveness of each component in the proposed DSSN method.

Effectiveness of contrastive Learning. In our study, we incorporate both low-level and high-level contrastive learning in our dual-level contrastive learning approach. Table 5 presents the results of our study. Without the use of both low-level contrastive and high-level contrastive, the accuracy was 78.33%. Using low-level contrastive alone results in a 0.57% improvement, while using high-level contrastive alone improves the accuracy by 0.86%. Notably, using both low-level and high-level contrastive further improves the accuracy by 1.25%, which shows the efficacy of our method.

Effectiveness of CPLG. As discussed in §3.4, the CPLG strategy considers difficulties of different classes and long-tailed classes, instead of using a fixed threshold during the pseudo-label generation. To test our method against a fixed threshold, we conduct experiments using a fixed threshold. Fig. 4 shows that our strategy outperforms using a fixed threshold of 0.96 and 0.92 since we set $r$ to 0.96 and $\tau_{\rm low}$ to 0.92 in CPLG. This finding further highlights the effectiveness of our proposed DSSN method. We chose these specific thresholds because, following our experiments, we establish 0.92 as the lowest threshold and used 0.96 as the factor for the maximum probability value. Additionally, Fig. 5 presents mIoU values of classes with long tails and those that are hard to learn during training, which demonstrates the effectiveness of CPLG strategy.

Qualitative Results. In Figs. 6 and 7, we present the qualitative results of our study on the PASCAL VOC 2012 validation set. DSSN is based on the DeepLab v3+ with ResNet-101 network and a 1/8 ratio. The integration of contrastive learning into our method improve the performance of our model for contour and ambiguous regions, while also enhancing the accuracy of some scenarios, as illustrated in Fig. 6. Furthermore, our proposed CPLG achieved substantial precision in certain classes that are typically challenging to learn, as illustrated in Fig. 7.