Learning to Count in the Crowd from Limited Labeled Data

Since the labeled dataset $\mathcal{D_{L}}$ comes with annotations, we employ $L_{2}$ error between the predictions and the GTs as supervision loss for training the network. This loss objective is defined as follows:

where $y^{pred}_{l}=g(z_{l},\phi_{d})$ is the predicted output, $y_{l}$ is the ground-truth, $z=h(x,\phi_{e})$ is the intermediate latent space vector. Note that, the subscript $l$ in the above quantities indicate that these are defined for labeled data.

Along with performing supervision on the labeled data, we additionally save feature vectors $z_{l}^{i}$’s from the intermediate latent space in a matrix $F_{z_{l}}$. Specifically, $F_{z_{l}}=\{z_{l}^{i}\}_{i=1}^{N_{l}}$. This matrix is used for computing the pseudo-GTs for unlabeled data at a later stage. The dimension of $F_{z_{l}}$ matrix is $N_{l}\times M$. Here, $M$ is the dimension of the latent space vector $z_{l}$. In our case, the latent space vector dimension is $64\times 32\times 32$ (see supplementary material for more details), which is reshaped to $1\times 65,536$. Hence, $M=65,536$.

Since the unlabeled data $\mathcal{D_{U}}$ does not come with any GT annotations, we estimate pseudo-GTs which are then used as supervision for training the network on unlabeled data. For this purpose, we model the relationship between the latent space vectors of the labeled images $F_{z_{l}}$ along with the corresponding GT $T_{y_{l}}$ and unlabeled latent space vectors $z^{pred}_{u}$ jointly using GP.

Considering the large dimensionality of the latent space vector, $K(F_{z_{l}},F_{z_{l}})$ can grow quickly in size especially if the number of labeled data samples $N_{l}$ is high. In such cases, the computational and memory requirements become prohibitively high. Additionally, all the latent vectors may not be necessarily effective since these vectors correspond to different regions of images in terms of content and size/density of the crowd. In order to overcome these issues, we use only those labeled vectors that are similar to the unlabeled latent vector. Specifically, we consider only $N_{n}$ nearest labeled vectors corresponding to an unlabeled vector. That is, we replace $F_{z_{l}}$ by $F_{z_{l},n}$ in Eq. (7)-(9). Here $F_{z_{l},n}=\{z_{l}^{j}:z_{l}^{j}\in nearest(z_{u}^{k},F_{z_{l}},N_{n})\}$, and $T_{y_{l},n}=\{y^{j}_{l}:z_{l}^{j}\in nearest(z_{u}^{k},F_{z_{l}},N_{n})\}$ with $nearest(p,Q,N_{n})$ being a function that finds top $N_{n}$ nearest neighbors of $p$ in $Q$.

The pseudo-GT for unlabeled data sample is given by the mean predicted in Eq. (8), i.e, $y_{u,pseudo}^{k}=\mu_{u}^{k}$. The $L_{2}$ distance between the predictions $y^{k}_{u,pred}=g(z_{u}^{k},\phi_{e})$ and the pseudo-GT $y_{u,pseudo}^{k}$ is used as supervision for updating the parameters of the encoder $f_{e}(\cdot,\phi_{e})$ and the decoder $f_{d}(.,\phi_{d})$.

Furthermore, the pseudo-GT estimated using Eq. (8) may not be necessarily perfect. Errors in pseudo-GT will limit the performance of the network. To overcome this, we explicitly exploit the variance modeled by the GP. Specifically, we minimize the predictive variance by considering Eq. (9) in the loss function. As discussed earlier, using all the latent space vectors of labeled data may not be necessarily effective. Hence, we minimize the variance $\Sigma_{u,n}^{k}$ computed between $z^{k}_{u}$ and the $N_{n}$ nearest neighbors in the latent space vectors using GP. Thus, the loss function during the unlabeled stage is defined as:

where $y^{k}_{u,pred}$ is the crowd density map prediction obtained by forwarding an unlabeled input image $x_{u}^{k}$ through the network, $y_{u,pseudo}^{k}=\mu_{u}^{k}$ is the pseudo-GT (see Eq. (8)), and $\Sigma_{u,n}^{k}$ is the predictive variance obtained by replacing $F_{z_{l}}$ in Eq. (9) with $F_{z_{l},n}$.

We combine the supervised loss Eq. (5) and unsupervised loss Eq. (11) to obtain the final objective function as follows:

where $\lambda_{un}$ is a hyper-parameter that weighs the unsupervised loss.

In this section, we conduct experiments in the semi-supervised settings by reducing the amount of labeled data available during training. The rest of the samples in the dataset are considered as unlabeled samples wherever applicable. In the following sub-sections, we present comparison of the proposed method in the 5% setting with other recent methods. For comparison, we used 4 datasets: ShanghaiTech (SH-A/B)[60], UCF-QNRF [10], WorldExpo [58] and UCSD [4]. This is followed by a detailed ablation study involving different architectures and various percentages of labeled data used during training. For ablation, we chose ShanghaiTech-A and UCF-QNRF datasets since they contain a wide diversity of scenes and large variation in count and scales.

For all the methods (except oracle), we limited the labeled data used during training to 5% of the training dataset. Rest of the samples were used as unlabeled samples. We used ResNet-50 as the encoder network. The results of the experiments are shown in Table 1. For all the experiments that we conducted, we report the average of the results for 5 trials. The standard deviations are reported in the supplementary. We make the following observations for all the datasets: (i) Compared to using the entire dataset, reducing the labeled data during training (to 5%) leads to significant increase in error. (ii) The proposed GP-based framework is able to reduce the performance drop by a large margin. Further, the proposed method achieves an average gain (AG)¹¹1$AG=\frac{G_{mae}+G_{mse}}{2}$, $G_{mae}=\frac{mae_{(\mathcal{D_{U}+D_{L}})}-mae_{(\mathcal{D_{L}})}}{mae_{(\mathcal{D_{L}})}}$, $G_{mse}=\frac{mse_{(\mathcal{D_{U}+D_{L}})}-mse_{(\mathcal{D_{L}})}}{mse_{(\mathcal{D_{L}})}}$ of anywhere between 10%-22% over the $\mathcal{D_{L}}$-only baseline across all datasets. (iii) The proposed method is able to leverage the unlabeled data more effectively as compared to Liu et al.[22]. This is because the authors in [22] using a ranking loss on the unlabeled data which is based on the assumption that sub-image of a crowded scene is guaranteed to contain the same or fewer number of people compared to the entire image. We observed that this constraint is satisfied naturally for most of the unlabeled images, and hence it provides less supervision (see supplementary material for a detailed analysis).

Recently, Wang et al.[55] proposed a synthetic crowd counting dataset (GCC) that consists of 15,212 images with a total of 7,625,843 annotations. The primary purpose of this dataset is to reduce the annotation efforts by training the networks on the synthetic dataset, thereby eliminating the need for labeling. However, due to a gap between the synthetic and real-world data distributions, the networks trained on synthetic dataset perform poorly on real-world images. In order to overcome this issue, the authors in [55] proposed a Cycle-GAN based domain adaptive approach that additionally enforces SSIM consistency. More specifically, they first learn to translate from synthetic crowd images to real-world images using SSIM-based Cycle-GAN. This transfers the style in the synthetic image to more real-world style. The translated synthetic images are then used to train a counting network (SFCN) that is based on ResNet-101 architecture.

While this approach improves the error over the baseline methods, its performance is essentially limited in the case of large distribution gap between real and synthetic images. Moreover, the authors in [55] perform a manual selection of synthetic samples for training the network. This selections ensures that only samples that are closer to the real-world images in terms of the count are used for training. Such a selection is not feasible in the case of unsupervised domain adaptation where we have no access to labels in the target dataset.

The proposed GP-based framework overcomes these drawbacks easily and can be extended to the synthetic-to-real transfer setting as well. We consider the synthetic data as labeled training set and real-world training set as unlabeled dataset, and train the network to leverage the unlabeled dataset. The results of this experiment are reported in Table 4. We used the same network (SFCN) and training process as described in [55]. As it can be observed, the proposed method achieves considerable improvements compared to the recent approach. Since we estimate the pseudo-GT for unlabeled real-world images and use it as supervision directly, the distribution gap that the network needs to handle is much lesser. This results in better performance compared to the domain adaptive approach [55]. Unlike [55], we train the network on the unlabeled data and hence, we do not need to perform any synthetic sample selection. Fig. 6 and 7 show sample qualitative results on the ShanghaiTech-A and UCF-QNRF datasets for the synthetic-to-real transfer protocol. The proposed method is able to predict the density maps more accurately as compared to the baseline.

Here, we provide details of the encoder and decoder architecture for all the experiments.

For all the above networks, the features of the final encoder layer are forwarded through a $1\times 1$ conv layer to reduce the dimensionality to 64 channels. The output of this $1\times 1$ conv is the feature embedding in the latent space which is used in GP modeling. Since the train crop size is $256\times 256$, the intermediate feature maps in the latent space is of dimension $64\times 32\times 32$.

In case of learning from the synthetic data, since we use the same network as in [55], all the layers after the dilated conv layers are used as decoder.

In this section, we provide details of the different datasets used for evaluating the proposed method in the main paper.

In this section, we study the effect of $\lambda_{un}$ on the overall performance. $\lambda_{un}$ weighs the unsupervised loss function in the Eq. 12 of main paper. For this study, we use the ShanghaiTech A dataset, due to its wide variety of scenes and diversity in the count. We conducted this experiment for the 5% data setting where 5% of the data was used as labeled data and rest was used as unlabeled data. We used Res50 encoder. Note that we perform the evaluation on the held-out validation set (and not on the test set). The results for different values of $\lambda_{un}$ are shown in Table 5.

We observed that the performance peaks when the value of $\lambda_{un}$ is $0.6$. $\lambda_{un}=0$ corresponds to only labeled data. This is the baseline performance. As we increase $\lambda_{un}$, we observe that the error improves. However, for $\lambda_{un}>0.6$, we see a small drop. This is because the network would not have learned to optimal level at the initial stages of training. Due to this the pseud-GT will be erroneous, and hence, using high weight for unsupervised at initial stages prohibits the network from reaching optimal performance.

Based on this experiment, we use $\lambda_{un}=0.6$ for all the experiments.

In this section, we conducted additional architecture ablation experiments using two recent crowd counting techniques: CSRNet [19] and Res101-SFCN [55]. WE use the 5% data-setting, where we use 5% of the data as labeled and rest as unlabeled. We evaluated both these methods on ShanghaiTech-A (SH-A) and UCF-QNRF datasets.For CSRNet, we use the layers upto the last dilated conv as the encoder. For the decoder, we use 2 conv layers as described earlier.

The results of this experiment are shown in Table 6. In addition to MAE/MSE, we rerport Average Gain (AG)²²2$AG=\frac{G_{mae}+G_{mse}}{2}$, $G_{mae}=\frac{mae_{(\mathcal{D_{U}+D_{L}})}-mae_{(\mathcal{D_{L}})}}{mae_{(\mathcal{D_{L}})}}$, $G_{mse}=\frac{mse_{(\mathcal{D_{U}+D_{L}})}-mse_{(\mathcal{D_{L}})}}{mse_{(\mathcal{D_{L}})}}$. We observed consistent gains in both the cases when we used the proposed GP-based method to leverage unlabeled data.

In this section, we report the standard-deviations for the experiments with our proposed method corresponding to Table 1 and Table 4 in the main paper. See Table 7 and Table 8. Note that the standard deviations are computed using 5 trials.