DAOT: Domain-Agnostically Aligned Optimal Transport for Domain-Adaptive Crowd Counting

It is well-known that optimal transport (OT) (Peyré et al., 2019; Panaretos and Zemel, 2019) has been widely applied in domain adaptation tasks, particularly in generating target domain data by finding the optimal distance between the source and target domain feature distributions. However, most of these methods (Damodaran et al., 2018; Xu et al., 2020; Chai et al., 2022) focus on generating images similar to those in the target domain, which requires finding the optimal distance between the generated and target images. In contrast, the proposed DAOT differs from traditional methods of OT in domain adaptation, as we establish correspondence between source and target domain distributions without generating new images or altering source domain images, similar to a feature-matching task (Jin et al., 2021).

Compared to point-to-point matching methods (Sarlin et al., 2020; Han et al., 2022; Wang et al., 2023), the proposed DAOT has a broader scope as it involves region-to-region matching. Moreover, we use a distance measurement method that emphasizes spatial structure, namely SSIM. Note that we do not perform person-level matching, as opposed to DR.VIC (Han et al., 2022). Our focus is not on the existence of a matching relation between the query and gallery, but on specific matching and transfer relations.

In crowd counting, OT has been used as a loss function, with DM-Count (Wang et al., 2020b) being one of the first proposed methods. Several methods (Song et al., 2021; Lin and Chan, 2023) have adopted this type of loss. Other than loss functions, we use OT to find the best match rather than simply measuring the difference.

Most UDA crowd counting methods (Wang et al., 2019b, a; Gao et al., 2021; Liu et al., 2023b; Xie et al., 2023) focus on classifying individual images into either the source or target domain, and therefore aim to bridge the gap between the source and target domains on a per-image basis. However, in Real-Real domain adaptation tasks, many images may have greater intra-domain differences than inter-domain ones. Addressing only the differences within individual images may not be sufficient to fundamentally bridge the gap between domains. Instead, we propose a domain adaptation method based on aligning the overall distribution of data in the source and target domains, rather than focusing solely on individual images. Several crowd counting methods (Zou et al., 2021; Zhu et al., 2022) consider that there are similarities between source and target domains and thus adapt between domains by finding similarities. However, individual similarity retrieval has shifted. Perhaps the found source and target distributions do have strong similarities, but due to the existence of retrieval bias, much of the information in the source distribution is not fully utilized, and only the distribution with the highest similarity is left. In essence, this approach is unable to fully simulate the entire data domain, while the proposed DAOT seeks to maximize the search for transfer relation between the two distributions. Apart from the methods that seek to find the similarities between domains, other methods (Wu et al., 2021; Du et al., 2022) attempt to find common knowledge between them. However, in our opinion, many images in different domains differ significantly in density, scale, and perception, making it difficult to find common knowledge. Even within the same domain, it is challenging to find common knowledge. We believe that rich data is essential for the model to learn knowledge and to have stronger generalization abilities. Therefore, instead of seeking common knowledge, we bridge the domain gap by aligning domain-agnostic factor misalignment between the source and target domains.

In this paper, we consider domain adaptation as a problem of aligning domain-agnostic factors between domains, and we employ a region-level distribution-perception method, where the distribution is used to describe the domain-agnostic factors. As shown in Fig. 3, the source domain region is set to $S=x_{i},i\in(1,n)$, and the target domain region is set to $T=x_{j},j\in(1,m)$. The model trained with labeled data in the source domain is defined as $M_{S}$, the initial model in the target domain is defined as $M_{T0}$, with the training labels using the pseudo-labels predicted by the source domain model on the target domain data set. And the regions in the two domains are distribution-perception to obtain the source domain distribution $D^{S}=d^{S}_{i}$ and the target domain distribution $D^{T}=d^{T}_{j}$, respectively. In Sec. 3.2, the distance matrix $C$ is calculated using SSIM to measure the distance between each distribution and extended to form the cost matrix $\overline{C}$, and the final obtained simulated distribution in Sec. 3.3 is defined as $D^{M}=D^{S}_{r,j},\{r\}\subseteq\{i\},i\in(1,n),j\in(1,m)$. The main method for domain alignment is optimal transport (OT).

The principle behind OT is to minimize the cost of transporting one distribution to another, where the cost is usually defined as the distance between the two distributions. The OT problem can be formulated as a linear programming problem, where the objective function is to minimize the cost of transportation subject to some constraints. Therefore, the first step is to obtain a probability transition matrix that expresses the transfer between two distributions, called the cost matrix $\overline{C}$. Then, based on the cost matrix, the Sinkhorn algorithm (Sinkhorn and Knopp, 1967; Cuturi, 2013; Peyré et al., 2019) to find the optimal solution $\overline{P}$ solved to find the optimal solution for transporting between the distributions. In our paper, local distribution matching can be achieved by solving an augmented version of Kantorovich’s OT problem (Peyré et al., 2019). And $U(\overline{A},\overline{B})$ denotes the discrete of $\overline{A}$ and $\overline{B}$.

where $\overline{P}$ is a transportation matrix. $\overline{C}_{i,j}$ represents the distance between each element in the source and target domains, in Eq. (2).

We believe that domain-agnostic factors are variable in both inter-domains and intra-domain, which makes individual images not exclusively belong to a particular domain, while misalignment of these factors at the domain level lead to the domain gap. Therefore, this paper aims to bridge the domain gap by aligning domain-agnostic factors. To achieve domain-level alignment, we measure individual-level differences using the following algorithm.

In this paper, the inter-domain adaptation task is transformed into an inter-domain domain-agnostic factor alignment problem, and further, the alignment of domain-agnostic factors is considered as finding the optimal mapping between domain-level distributions, which are used to describe domain-agnostic factors at the domain level. It is also the solution of Eq. (1). The function we use to find the optimal transport is Sinkhorn (Sinkhorn and Knopp, 1967). The ultimate goal of the OT problem is to find a matrix $\overline{P}$ that optimally transfers the source distribution to the target distribution so that the product of $\sum{(\overline{P}\times\overline{C})}$. This is essentially a linear programming problem Eq. (1) with $n+m+2$ constraints Eq. (4), the constraints ensure that the probabilities of mapping between the source and target distributions sum up to one for each element, and vice versa.

Shanghai Tech (Zhang et al., 2016) is a benchmark for crowd counting contains 1,198 scene images and 330,165 labeled head positions, which is subdivided into PartA (A) and PartB (B). A is densely populated and contains 300 training images and 182 testing images; B is relatively sparser and contains 400 training images and 316 testing images.

UCF-QNRF (Q) (Idrees et al., 2018) is a challenging crowd counting dataset with rich scenes and varying viewpoints, densities, and illumination conditions. It contains 1,535 images of dense crowd scenes, with 1,201 training images and 334 testing images.

NWPU-Crowd (N) (Wang et al., 2020a) has 5,109 high resolution images and annotated entities 2,133,238. The large range of annotated entity numbers for a single image is extremely challenging. Additionally, some images have object annotation number of 0.

JHU-CROWD++ (J) (Sindagi et al., 2020) contains 4,372 images with a total of 1.51 million annotations, including some based on extreme weather variation and light variation, and introduces negative samples of entities annotated with zero.

Loss Function. We use the loss function proposed in FIDT (Liang et al., 2021) for training, where $\eta$ is set to 1 following FIDT.

Evaluation Metrics. In this paper, the evaluation metrics used are mean absolute error (MAE) and root mean square error (RMSE) introduced in (Zhang et al., 2016). MAE assesses model accuracy, while RMSE evaluates model robustness.

Optimal Transport. In A2Q setting, the regularization parameter $\epsilon$ is set to $e{-2}$, while in Q2B setting, it is set to $e{-1}$. All other settings are set to $e{-3}$. The algorithm stops at $T_{\max}=1000$.

Retraining. We use adaptive moment estimation (Adam) (Kingma and Ba, 2015) to optimize the model with the learning rate of $e{-5}$, and the weight decay is set as $5e{-4}$. We set the size of the training batch to $16$.

To evaluate the effectiveness of DAOT, this subsection compares it with other state-of-the-art cross-domain methods. We conduct quantitative evaluations through five sets of adaptation experiments, reporting the error between the predicted density maps and ground-truth maps on the target set for each dataset pair. We compare DAOT with three types of counting methods: 1) state-of-the-art counting methods without domain adaptation (Xu et al., 2019; Cheng et al., 2021), 2) Syn-Real domain adaptation methods (Zhu et al., 2017; Wang et al., 2019a; Sindagi et al., 2020; Gao et al., 2021; Gong et al., 2022; Liu et al., 2022), and 3) Real-Real domain adaptation methods (Zou et al., 2021; Wu et al., 2021; Du et al., 2022; Zhu et al., 2022), as well as our baseline method, FIDT (Liang et al., 2021). Notably, the source domains for Syn-Real methods are large-scale synthetic GCC (Wang et al., 2019a).

As shown in Table 1, the proposed DAOT achieves the best performance in five groups’ experiments (A2B, A2Q, N2A, N2B, and J2B) and obtained the second-best performance among the other groups (Q2A, Q2B, and J2Q). Especially with a significant improvement over other methods in the setting of A2B and N2B. The DAOT outperforms the baseline in all 8 settings and achieves better results than the similarity-based FGFD (Zhu et al., 2022) method in multiple settings. Compared with D2CNet (Cheng et al., 2021), the proposed DAOT is slightly lower than D2CNet, indicating that D2CNet is more suitable for the group with the source domain as Q, while the DAOT has a stronger generalization ability. Compared with C2MoT (Wu et al., 2021), the DAOT is slightly lower than C2MoT, but the amount of data used for retraining is much lower than that of C2MoT, and C2MoT is more dependent on large amounts of data. When the target domain is B, DAOT often achieves better performance, which proves that it has a stronger fitting ability for sparse density distribution. In the setting of A2Q and N2B with aligned domain-agnostic factors, the proposed DAOT method can further improve the excellent performance.

Ablation Study on Region Size The transition relation between entire images across domains is limited, while the transition relation between regions exists extensively across different domains. As the region size decreases, the accuracy of distance measurement between distributions will decrease. As the region size increases, the transition relation between distributions gradually decreases. As shown in Fig. 5, as the region size decreases from the entire image to 256, the MAE and RMSE metrics correspondingly improve. However, as the region size continues to decrease beyond 256, MAE and RMSE begin to exhibit a rising trend.

Ablation Study on Various Factors Performance To achieve domain adaptation in crowd counting, we transform it into a domain-agnostic factors alignment problem and adopt optimal transport (OT) to find the best transfer distance between distributions. In Table 2, using all factors achieves the best performance in all experiments. OT significantly improved performance, especially in Q2A experiment. Using only pseudo-labels and SSIM achieves the second-best performance, showing a slight improvement over using only source-domain data. This suggests that adjusting the density distribution misalignment can bridge the domain gap.

Ablation Study on Distance Metric This paper aims to find the optimal transport between distributions, by measuring the distribution distance between the source and target domains before using OT. The OT ultimately obtains a more reasonable distribution distance. The distance measurement method is crucial for the final solution, and we conduct ablation studies on different similarity metrics, which are shown in Table 3. The three distance metric methods have all shown some improvement compared to using source domain data alone, especially when using SSIM.

Analysis of t-SNE Visualization To more intuitively demonstrate the changes in domain-agnostic factors before and after domain adaptation, we visualize Fig. 6. In A2B, the clustering of the overall image distribution shows the most obvious difference, with the distribution of B more concentrated in the lower left corner. This indicates that there is more similarity in the crowd distribution between A and B, but the range of A is wider. In the regional clustering, most of the data in the regions of A and B overlap, but the distribution range of A is still wider than that of B. From the third and fourth columns, it can be seen that although the simulated distribution comes entirely from the source domain, it is highly overlapped with the target domain and more concentrated than the distribution of the source domain.

In A2Q, the clustering of the overall image distribution shows higher overlap, but the distribution of Q is more concentrated in the center. In regional clustering, the distribution of Q shows a more obvious clustering phenomenon. In the visualizations of the third and fourth columns, it can be clearly seen that the simulated distribution is closer to the target domain.

Analysis of Cross-Domain Density Map Prediction To visualize the cross-domain performance of different methods in a straightforward manner, we visualize Fig. 7. The figure illustrates that the proposed DAOT method outperforms other methods in various scenarios, ranging from dense to sparse (B2A), from large-scale to small-scale (Q2A), and even from small-scale dense to large-scale (A2Q). Particularly in Q2A setting, the DAOT prediction is very close to the ground truth. In the first two rows (Q2A and A2Q), there are fewer erroneous classifications of the background, and in the third row (A2B), the estimation of the crowd is more accurate. We have highlighted these parts with red boxes.

Analysis of Distribution Match The final output distribution has a matching relation with the initial target domain distribution, and we visualize the distribution matching in Fig. 11. We demonstrate the retrieved source domain regions using different methods on the target domain regions ranging from dense to sparse. As shown in the figure, the DAOT achieves superior performance across all density levels, even in scenarios with few people or only background. Compared with the method using only SSIM, the proposed DAOT achieves better performance in the second column with denser scenarios, and has better perception simulation for the densest scenario. Compared with the diffusion-based FGFD (Zhu et al., 2022), it achieves better results in the densest and sparsest scenarios, and has better simulation for the location of pedestrians.