Spatio-Temporal Pixel-Level Contrastive Learning-based Source-Free
Domain Adaptation for Video Semantic Segmentation

Spatio-temporal fusion block. Next, we propose a spatio-temporal fusion block $F$ to extract spatio-temporal feature representations from the previous and the current features $z_{t-1}$ and $z_{t}$ (see Figure 3 (a)). It adopts the estimated optical flow $o_{t-1\to t}$ to warp the previous feature $z_{t-1}$ to the propagated feature as: $z^{\prime}_{t-1}=W(z_{t-1};o_{t-1\to t})$, where $W$ denotes the warping operation. This feature propagation aligns the pixel correspondence between the previous and the current features, which is crucial for the dense prediction task. Then a fusion operation $f$ is used to fuse the cross-frame features into a spatio-temporal feature as: $z_{(t-1,t)}=f(z^{\prime}_{t-1},z_{t})$.

Cross-frame augmentation. Meanwhile, we perform cross-frame augmentation [49] that applies randomized spatial transformations $T$ on each input frame to generate an augmented video sequence: $\tilde{X}=T(X)=\{\tilde{x}_{t-1},\tilde{x}_{t}\}$. Then we apply the same spatio-temporal feature extraction process on $\tilde{X}$ and extract the augmented spatio-temporal feature $\tilde{z}_{(t-1,t)}$. The augmentation $T$ contains randomized Gaussian blurring and color jittering transformations.

Pseudo pixel-wise feature separation. STPL aims to acquire pixel-level representations that are similar among the same-class pixel samples but distinct among different-class pixel samples. Since we do not have target domain labels, we use our VSS model’s prediction for the input $X$ as pseudo-label $\hat{y}$. Subsequently, we use $\hat{y}$ to do pixel-wise feature separation. To maintain high-quality pseudo-labels, we set a hyperparameter of confident proportion $k$ to control the proportion of pixels preserved as pseudo-labels. More precisely, the confident pseudo-labels $\hat{y}^{*}$ are obtained by $\hat{y}^{*}=topk(\hat{y};k)\subset\hat{y}$, where $topk$ is an operation that returns the $k$-proportion of the most confident predictions according to their probability scores.

Pixel-to-pixel contrastive loss. To perform CL, we first adopt a projection head $H$ to project our feature representations $z^{h}_{(t-1,t)}=H(z_{(t-1,t)})$ and $\tilde{z}^{h}_{(t-1,t)}=H(\tilde{z}_{(t-1,t)})$, similar to SimCLR [5]. According to the generated confident pseudo-labels $\hat{y}^{*}$, we denote the confident pixel representation sets in $z^{h}_{(t-1,t)}$ and $\tilde{z}^{h}_{(t-1,t)}$ as $z^{*h}_{(t-1,t)}\subset z^{h}_{(t-1,t)}$ and $\tilde{z}^{*h}_{(t-1,t)}\subset\tilde{z}^{h}_{(t-1,t)}$, respectively. Next, consider a query confident pixel representation $q\in z^{*h}_{(t-1,t)}$ (i.e., $q$ is a pixel representation in the feature $z^{*h}_{(t-1,t)}$) with a predicted pseudo-label $\hat{y}^{*}_{q}$, we define its positive pair set as:

$$P_{q}\equiv\{q^{+}\in\tilde{z}^{*h}_{(t-1,t)}:\hat{y}^{*}_{q^{+}}=\hat{y}^{*}_{q}\},$$

(1)

i.e., all the same-class pixels in the augmented feature $\tilde{z}^{*h}_{(t-1,t)}$. Then we define its negative pair set as:

$$N_{q}\equiv\{q^{-}\in\tilde{z}^{*h}_{(t-1,t)}:\hat{y}^{*}_{q^{-}}\neq\hat{y}^{*}_{q}\},$$

(2)

i.e., all the different-class pixels in $\tilde{z}^{*h}_{(t-1,t)}$. We follow SupCon [20] to develop a CL scheme with multiple positive pairs. The complete formulation of the proposed STPL contrastive loss is as follows:

$$\mathcal{L}^{stpl}_{q}=\frac{-1}{|P_{q}|}\sum\limits_{q^{+}\in P_{q}}\log\frac{\exp(q\cdot q^{+}/\tau)}{\sum_{q^{-}\in N_{q}}\exp(q\cdot q^{-}/\tau)},$$

(3)

where $\tau$ is a temperature parameter, and the $\cdot$ symbol denotes the inner product. Finally, the overall objective for the given video sequence input $X$ is defined as:

$$\mathcal{L}^{stpl}=\frac{1}{|z^{*h}_{(t-1,t)}|}\sum\nolimits_{q\in z^{*h}_{(t-1,t)}}\mathcal{L}^{stpl}_{q}.$$

(4)

This objective enforces the pixel representations in the original spatio-temporal features to be similar to that of the same-class pixels in the augmented features, while being distinct from that of the different-class pixels. This explicitly learns semantic correlations among pixels in the spatio-temporal space and thus can achieve better class discriminability. The proposed STPL provides a strong self-supervision for video adaptation under the SFDA setup.

Spatial-only contrast. Let us turn off the fusion operation $F$ of the STPL framework with an identity operation. Then, let us allow only the current frame feature $z_{t}$, and similarly, only the augmented current frame feature $\tilde{z}_{t}$ to pass through the fusion block. After the projection head and confident filtering steps, the contrastive loss would be computed between $z^{*h}_{t}$ and $\tilde{z}^{*h}_{t}$ instead of the spatio-temporal features $z^{*h}_{(t-1,t)}$ and $\tilde{z}^{*h}_{(t-1,t)}$. That is, in Eq. (3) and Eq. (4), it becomes that $q\in z^{*h}_{t}$ and $\{q^{+},q^{-}\}\in\tilde{z}^{*h}_{t}$. This computes contrast between only spatial variations and thus is a spatial-only special case of STPL. We denote this loss as $\mathcal{L}^{spa}$.

Temporal-only contrast. Let us consider a duplicate copy of the input video as an augmentation (i.e., $\tilde{X}=X$). Next, let us turn off the fusion operation $F$ of STPL, allowing only the current frame feature $z_{t}$ and the augmented previous frame feature $\tilde{z}_{t-1}$ to pass through the fusion block. Here $\tilde{z}_{t-1}=z_{t-1}$ since $\tilde{X}=X$. Hence, after the projection head and confident filtering steps, the contrastive loss would be computed between $z^{*h}_{t}$ and $z^{*h}_{t-1}$. That is, in Eq. (3) and Eq. (4), it becomes that $q\in z^{*h}_{t}$ and $\{q^{+},q^{-}\}\in z^{*h}_{t-1}$. This computes contrast between only temporal variations and thus is a temporal-only special case of STPL. We denote this loss as $\mathcal{L}^{tem}$.

Naïve combination. To learn spatio-temporal contrast, a naïve way would be to combine the spatial-only and temporal-only contrastive losses together: $\mathcal{L}^{spa}+\mathcal{L}^{tem}$. Our experiments in Sec. 4.3 show that the naïve combination is sub-optimal compared to STPL. This demonstrates that the proposed STPL is a non-trivial unified spatio-temporal framework. Figure 4 compares the proposed spatio-temporal contrast $\mathcal{L}^{stpl}$, spatial-only contrast $\mathcal{L}^{spa}$, and temporal-only contrast $\mathcal{L}^{tem}$.

Datasets. We evaluate our method on two widely used domain adaptive VSS benchmarks: VIPER [36] $\to$ Cityscapes-Seq [7] and SYNTHIA-Seq [37] $\to$ Cityscapes-Seq. VIPER has 133,670 synthetic video frames with a resolution of 1080$\times$1920. SYNTHIA-Seq consists of 8,000 synthetic video frames with a resolution of 760$\times$1280. We consider VIPER and Synthia-Seq as source datasets to pre-train source models, respectively. Cityscapes-Seq is a realistic traffic scene dataset. It contains 2,975 training and 500 validation video sequences with a frame resolution of 1024$\times$2048. We use it as a target dataset. Following [12, 49], we resize the frames of VIPER and Cityscapes-Seq to 760$\times$1280 and 512$\times$1024, respectively. For evaluations, the output predictions are interpolated to the original size.

Implementation details. Following [12, 49], we employ ACCEL [19] as our VSS network. It includes two segmentation branches, an optical flow estimation branch, and a prediction fusion layer. These branches consist of DeepLabv2 [4] architecture with ResNet-101 [14] backbone, FlowNet [8], and a 1$\times$1 convolution layer, respectively. All the adaptation models are trained by an SGD optimizer with an initial learning rate of $2.5e^{-6}$ and a momentum of 0.9 for 20k iterations. The learning rate decreases along the polynomial decay with a power of 0.9. We set the temperature $\tau=0.07$ and the confident proportion $k=0.7$. The mean Intersection-over-Union (mIoU) is used as the evaluation metric. Our experiments are implemented using PyTorch [35].

Baselines. Since the proposed STPL is the first SFDA method for VSS, we compare it with multiple related domain adaptation state-of-the-art approaches described as follows. (1) Image-based UDA: FDA [51], PixMatch [33] and RDA [17]; (2) Image-based SFDA: UR [39] and HCL [16]; and (3) Video-based UDA: DA-VSN [12], VAT-VST [38] and TPS [49]. The image-based approaches are applied to videos by using a VSS backbone (ACCEL in our experiments), following the practice of [12, 49]. Furthermore, to fairly assess our STPL, we create the SFDA versions of these video-based UDA approaches as our (4) Video-based SFDA baselines. We remove all of their loss terms containing source data while keeping all the loss terms computed from only target data. We use the * symbol to denote these baselines. The results of the source-only and oracle (i.e., trained with target domain labels) models are also reported for reference. For fair comparisons, all four types of baselines use the same VSS backbone and training settings.

VIPER $\to$ Cityscapes-Seq. Table 1 reports the evaluation results on the VIPER $\to$ Cityscapes-Seq adaptation benchmark. The proposed STPL outperforms all four types of baselines by decent margins, which is 15.1% higher than the source-only model and 3.6% higher than the best-performing competitor. In particular, its superiority over the image-based SFDA approaches indicates the benefits of a video-based solution and demonstrates the effectiveness of our spatio-temporal strategy for videos. We can also observe that the video-based UDA approaches suffer from performance degradation when applied to SFDA. Whereas, STPL achieves better performance even compared to their UDA results relying on source data.

SYNTHIA-seq $\to$ Cityscapes-Seq. Table 2 provides the results on the SYNTHIA-Seq $\to$ Cityscapes-Seq benchmark. Similarly, our STPL is better than most baselines. Although TPS achieves the best accuracy under UDA, this requires accessing source data. Moreover, TPS*’s accuracy dramatically reduces to 22.1% under SFDA, showing that it is not a proper solution when source data are unavailable. Overall, these results clearly demonstrate the superiority of STPL.

Qualitative results. Figure 5 shows examples of qualitative results on VIPER [36] $\to$ Cityscapes-Seq [7]. The source-only model produces noisy and inconsistent predictions on the road and sidewalk, showing the domain shift effect. UR, an image-based SFDA method, suffers from inaccurate sky predictions and cannot detect the whole sidewalk. In contrast, the proposed STPL obtains more accurate segmentation results with high temporal consistency across the video sequence. This indicates the importance of a video-based strategy for the VSS task and demonstrates our method’s effectiveness. The qualitative and quantitative results are consistent.

Objective functions. We conduct an ablation study to validate the effectiveness of our spatio-temporal objective for adaptation. We create several variants for comparison. Vanilla Self-training simply computes the cross-entropy loss between predictions and pseudo-labels with a confident threshold. Duplicate CL computes the pixel-level contrastive loss between two identical video frames, i.e., the loss described in Sec. 3.2 but uses a duplicate copy as an augmentation and passes through the current frame features only. Temporal-only CL, Spatial-only CL and Naïve T+S CL are described in Sec. 3.3, whose objective functions are $\mathcal{L}^{tem}$, $\mathcal{L}^{spa}$ and $\mathcal{L}^{tem}+\mathcal{L}^{spa}$, respectively.

Fusion operations. As discussed in Sec. 3.1, our STPL framework is compatible with various fusion operations used to extract spatio-temporal features. Here we consider and compare different fusion operations, such as element-wise addition, 1$\times$1 convolution layer, concatenation, and the proposed STAM module. In Table 4, we can observe that STAM achieves the best performance, showing its effectiveness. On the other hand, adopting any fusion operation can outperform all the baselines in Table 1 and variants in Table 3. This demonstrates that STPL maintains superior performance regardless of the choice of fusion operations.

Feature visualization. Figure 6 provides the t-SNE visualization [42] of the feature space learned for the VIPER $\to$ Cityscapes-Seq benchmark. For simplicity, we sample four classes (road, traffic light, car, and bicycle) to visualize. Each point in the scatter plots stands for a pixel representation. We compute the intra-class variance $\sigma_{intra}$ (lower is better) and inter-class variance $\sigma_{inter}$ (higher is better) of the feature space to provide a quantitative measurement. As can be seen, TPS*, which is originally designed for UDA, has a less discriminative feature space under the SFDA setup. It obtains a higher $\sigma_{intra}$ and a lower $\sigma_{inter}$ than the source-trained model. HCL, an image-based SFDA approach, acquires a higher $\sigma_{inter}$, but its $\sigma_{intra}$ is much higher. In comparison, the proposed STPL learns the most discriminative feature space. Unlike HCL, STPL leverages spatio-temporal information for video adaptation, and the benefit is clearly reflected by the lowest $\sigma_{intra}$ and the high $\sigma_{inter}$. This demonstrates STPL’s ability to learn semantic correlations among pixels in the spatio-temporal space.

Feature space neighborhood. This analysis inspects the neighborhood of the feature space learned by the proposed STPL, which quantitatively measures the discriminability of a feature space [50]. We randomly select several video samples and extract the features at the pixel level. For an unbiased analysis, 500 pixel representations are considered for each semantic class to create a feature analysis set. Next, we query each representation in the set and retrieve the $k$-nearest neighbors of that representation. Among the retrieved $k$ nearest representations, we inspect the percentage of the same-class representations it contains.