A Strong Baseline for Crowd Counting and Unsupervised People Localization

Crowd counting is a computer vision task which aims to output a density map indicating the distribution of crowd and get the estimated count of people by calculating the integral over the map in still images. In recent years, many novel networks are designed and the MAE on ShanghaiTechA dataset has been improved from 110.2(MCNN) to 61.7(SPN). It is important to note, however, that the improvement is not only the result of newly proposed methods. It also benefits from various training tricks in implementation. According to He et al. (2019) training details including data preprocessing, changes of loss function, the ground truth generation, and learning objectives setting make a big difference.

Since Li et al. (2018), most researchers extend their work using VGG-16 as the backbone, and the main components of the fully convolutional networks are like the front-end and back-end in 1(a). Existing method Wang and Breckon (2019) has shown that training tricks can make a big difference. Besides, there are also some methods using other classification networks as the backbone. To make a fair comparison and construct a strong baseline easy to follow for the academia and the industry, we collect and evaluate a set of backbones and tricks for regressing a more accurate density map. As Figure 1(b) shows, by combining these tricks, the MAE can be decreased from 74.9 to 60.1, and RMSE can be decreased from 126.1 to 95.5, which is better than SPN. They show consistent improvement on multiple datasets which proves the effectiveness of this strong baseline.

Another drawback of existing previous methods is the lack of position information for each person. Note that position information is essential for us to understanding the crowd images, we extend our work to find the position of each person. Liu et al. (2019a) and Liu et al. (2019d) have combined detection networks to located heads. However, there are no standard labels in crowd counting dataset as in detection datasets, so the ground truth is not reliable. Besides, we have to retrain the parameters of detection networks for different datasets which tend to be time consuming in practical use. In fact, the estimated density maps have indicated the positions by showing clusters around each head center. A human can find most of the heads easily by observing the clusters. Based on this observation, we proposed an unsupervised localization method named isolated KMeans based on cluster algorithms. Because people count $n$ can be obtained by calculating the integral, the objective of this problem is simplified into finding $n$ centers indicating the position of $n$ persons. Each point whose value is bigger than 0 in density maps is regarded as a potential head and cluster centers converge to the points near the heads. In addition, isolated KMeans can locate the head centers under local count constraints to make the number of cluster centers and people match both globally and locally. We can integrate this localization method with any other existing crowd counting methods without training numerous parameters. To the best of our knowledge, we are the first to use unsupervised method to do people localization in this field.

To summarize, our main contribution is three-fold:

• •

  We collect a series of backbones and training tricks for crowd counting and evaluate their impacts on the model performance to validate the effectiveness and make a fair comparison.

• •

  We develop a strong baseline by combining the effective tricks which can improve the accuracy and density map quality significantly on three widely used datasets.

• •

  We propose an unsupervised method for people localization in density maps. It is achieved using a cluster algorithm named isolated KMeans which divides one density map into subregions and locate the heads under local people count constraints. This method frees us of training a large number of parameters of detection networks and can be integrated with any existing counting method.

All datasets only provide discrete point labels in the centers of heads, which is hard to learn. We firstly convert the labeled points into density maps with continuous distribution.

Ground Truth Density Map Generation. Following most previous works, each labeled point at a head’s center is substituted with a Gaussian distribution, and superimposing multiple Gaussian distributions produces the ground truth density map. This is formulated as

where $M_{i}$ means the density map, and $head_{j}$ is the coordinates of $j_{th}$ head annotation. $G(x)$ is the Gaussian distribution. $n$ is the number of head annotations. In ShanghaiTech PartB dataset, the standard deviation parameter $\sigma_{j}$ is set to 15. In other datasets, $\sigma_{j}$ is calculated as $\sigma_{j}=\beta*d_{j}$ where $d_{j}$ is the average distance of $x_{j}$ and its $k$ nearest neighbors. In this paper we adopt the configuration of $\beta=0.3$ and $k=3$.

Training Settings and Evaluation Metrics. Experiments are conducted on ShanghaiTech PartA dataset for comparison. We use PyTorch to implement all training process. For all models, the total training epochs is set to 200 and initial learning rate is set to 1e-5. Adam optimizer and ReduceOnPlateau learning rate scheduler are used to optimize the parameters. We adopt two mainly used metrics MAE and RMSE for evaluating the accuracy of counting and the quality of the density maps.

Backbones. Most existing methods adopt VGG-16 as the backbone. Some literature also adopts ResNet-50Wang et al. (2019) and InceptionV3Wang and Breckon (2019). Similarly to Li et al. (2018), we remove the fully connected layers and preserve the convolutional layers to compare the performance of these backbones and to explore if batch normalization should be used or not. For back-end, we use the configuration of $3\times CS(512,3,2)-Upsample-CS(256,3,2)-Upsample-CS(128,3,2)-Upsample-CS(64,3,2)-Upsample-C(1,1,1)$. $N\times CS(m,s,d)$ means N convolutional layers integrated with SE block Hu et al. (2018) with $m$ filters whose size is $s\times s$ and dilation rate is $d$ followed by the Swish activation layer by Ramachandran et al. (2017). We use bilinear interpolation to upsample the feature maps.

For models without batch normalization, we train them using original images of different sizes. For models with batch normalization, we train them using $256\times 256$ patches randomly cropped from the images. To be specific, we randomly choose 4 images at a time and 4 patches are randomly cropped from each of them to constitute a batch. Finally one batch consists of 16 patches of $256\times 256$. In experiments, if the back-end contains batch normalization layers, the model converge to output 0. So batch normalization layers only exist in front-end. The performance of different backbones is reported in Table 1. Due to the superior performance, we select first 13 layers of VGG-16 as the backbone and conduct following experiments.

Many loss functions are proposed to improve the quality of density maps. Among them, MSE loss ($\ell_{MSE}$) is the mostly used one for its simplicity and effectiveness. It is defined as:

Spatial Abstraction Loss ($SAL$) is proposed by Jiang et al. (2019) which progressively computes the MSE losses on multiple abstraction levels. The computation is formalized as:

where $k_{a}$ indicates the abstraction level and $K_{a}$ is set to 3. $\hat{M_{i}^{k_{a}}}$ and $M_{i}^{k_{a}}$ are the downsampled density maps by pooling layers and $N_{k_{a}}$ is the number of pixels within a map.

Multi-scale density level consistency loss ($\ell_{c}$) is proposed by Dai et al. (2019). $\ell_{c}$ loss separates the density maps into subregions using adaptive average pooling and calculates L1 loss to enforce the consistency at different scale levels. It is defined as:

where $s$ is the scale levels, and $k_{s}$ is size of density maps. They are set to 3 and $1\times 1,2\times 2,4\times 4$ respectively.

SSIM is proposed to measure the similarity of images byWang et al. (2004) and Local Pattern Consistency Loss ($\ell_{SSIM}$) is introduced by Cao et al. (2018) to enhance the structure consistency. A fixed-parameter kernel $W(p)$ to adopted to define the weights of different positions in one sliding window. For the same location $x$ in density maps $\hat{M_{i}}$ and $M_{i}$, the local statistics are defined as:

and the SSIM index and SSIM loss are calculated as:

We train the model using different loss functions, and the results are reported in Table 2.

As we can see, $\ell_{MSE}+\ell_{c}$ gets the best MAE performance. Also because of its simple form, we select it as the loss function.

We collect various tricks in training and evaluate each of them. By combining the effective tricks, we can boost the model’s accuracy without changing the architecture.

Data Augmentation. In crowd counting, estimated density maps are sensitive to the size of people. Resize and rotation is harmful to the performance of the network, so we apply cropping to enlarge the training set. There are 4 kinds of cropping: (i) random 0.3 - random0.9 means randomly cropping by ratio 0.3 - 0.9 from each image. (ii) fixed 0.5 means cropping 4 non-overlapping quarters from fixed locations in original images. (iii) fixed + random 0.5 means 4 patches are cropped from fixed locations and 5 patches randomly cropped from each image. (iv) mixed means randomly cropping by a random ratio in {0.3, 0.4, 0.5, 0.6, 0.7}. Table 3 shows that random 0.3 is the most effective data augmentation. In addition, cropping can also reduce the training time.

Curriculum learning. In Wang and Breckon (2019), the curriculum is designed based on the fact that dense crowds are more difficult to count than sparse crowd. In training process, the weights of areas with higher density are relative lower and gradually increased to equal to areas with lower density. This is implemented by a weight matrix $W(e)$ which is defined as:

$T(e)$ is a threshold matrix which is defined as:

where $k_{e}$ and $b_{e}$ are coefficients determined by prior knowledge and $e$ denotes the epoch number. $W$ has the same size as the density map $M_{i}$. Then $\ell_{MSE}$ is calculated as:

Empirically, we set $k_{e}=2e-3,b_{e}=5e-3$. Using curriculum learning can decrease MAE to 63.3 as Table 3 shows.

Value Expansion. The statistics of pixel values in mainly used datasets can be viewed in Figure 2. Many pixel values are very small which can lead to the loss of precision.

To facilitate the training process, we multiply the ground truth density map by a scale factor $k_{expand}$. In inference process, we can multiply the estimated density map by $\frac{1}{k_{expand}}$. Using value expansion, we can improve the MAE to 62.6.

Validate by Patch. Cao et al. (2018) firstly proposes the patch-based validation strategy. In validation process, each image is divided into several quarters, and quarters are fed into the network. By calculating the sum of each quarter’s output, we can get the overall count. Table 3 reports that RMSE can be improved to 95.4 using this strategy.

Besides density maps, Liu et al. (2019b) and Wang and Breckon (2019) propose using attention maps to emphasize the crowd regions and weaken the impact of background regions. An extra branch with the configuration of $2\times CS(64,3,2)-C(1,1,1)-Sigmoid$ is inserted after the penultimate layer and can produce a one-channel attention map $M^{att}_{i}$ with same size of density map $M_{i}$. This branch aims to regress an attention map whose value is close to 1 in foreground regions and close to 0 in background region and is illustrated in Figure 3.

Ground Truth Attention Map Generation. There are two ways of generating attention maps. (i) Window-based. We set a $25\times 25$ windows centered at each labeled point. Values within those windows are set to 1 and those out of them are set to 0. (ii) Threshold-based. We set the threshold at the value bigger than 40% of the values. We test the two ways, and way (i) shows a more accurate result.

Attention Map Loss. The loss function of attention map is defined as

The total loss is a weighted sum:$\mathcal{L}_{total}=\ell_{MSE}+\ell_{C}+\lambda\ell^{att}$. Empirically, when $\lambda=0.5$, the model gets best performance.

Density Map Size. We change the number of upsample layers to evaluate the impact of different density map sizes. By adding 1 - 4 upsample layers, we can get $\frac{1}{8}\times\frac{1}{8}$ - $1\times 1$ size of density maps. Table 4 shows that estimating the same size of density maps benefits the performance.

We evaluate our final model with other state-of-the-art methods on three mainly used datasets.

ShanghaiTech dataset is introduced by Zhang et al. (2016) which consists of PartA and PartB. PartA has 300 training images and 182 testing images with relatively high density. PartB has 400 training images and 316 testing images with relatively low density.As table 5 shows, our strong baseline decreases MAE by 2.6% on PartA and RMSE by 0.8% on PartB.

UCF-QNRF dataset is introduced by Idrees et al. (2018). It is has 1201 training images and 334 testing images with high resolution. Performance on this dataset is reported in Table 6, and our strong baseline decreases MAE by 2.7%.

UCF_CC_50 dataset is introduced by Idrees et al. (2013) including 50 crowd images with extremely high density. We follow the standard 5-fold cross-validation. It can be viewed in Table 6 that our baseline decreases the MAE by 4.0%.

In this paper, we collect and evaluate a series of backbones and training tricks in training a neural network for crowd counting. By selecting the best backbone and applying effective training tricks together, we construct an efficient and accurate baseline which improve the MAE and RMSE significantly on three mainly used datasets. We also propose an unsupervised people localization method named isolated KMeans. This clustering algorithm uses the point set constructed from the density maps which eliminates the time of training detection networks, and can also be integrated with any existing counting method.