Fine-Grained Crowd Counting

Crowd counting algorithms aim to estimate the number of people in a whole image via either direct regression methods or density map based methods. Traditional methods count the number of people by detecting individuals, either by their whole body or body parts [2]. However, detection approaches will fail on dense crowds with many occlusions. To bypass detection, direct regression methods were proposed to directly predict the number of people from low-level features [3, 4]. [5] proposes to estimate the number of people using multiple features to improve counting performance. However, due to scale variation and occlusion, the performance of direct regression methods is also limited.

Density map approaches use a density map as an intermediate representation, which is then summed to get the count over a region. In recent years, density map methods achieve better performance than direct regression since the spatial distribution of the crowd is taken into consideration. The typical design of density map estimation methods can be divided into two steps: 1) density maps are generated from dot map annotations; 2) deep learning models are designed to estimate density maps from images. The most popular method to generate a density map is to convolve the dot map with a fixed or adaptive Gaussian kernel. However, since hand-crafted density maps may not be optimal in the sense of end-to-end learning, [6] proposes a learning method to generate customized density maps for different networks/datasets. [7] proposes a Bayesian loss to evaluate the error between the dot ma and predicted density map.

Various deep architectures for estimating density maps have been proposed to handle scale variations, refine results, perform domain adaptation, or exploit context information. To handle scale variation, [8] proposes a multi-column network (MCNN) that extracts multi-scale features by three branches with different receptive field sizes and fuses the multi-scale features together, while [9] proposes to select between the branches. SANet [10] proposes a scale aggregation module applied to all convolutional layers, while [11] proposes a tree-structured CNN to deal with scale variations. [12] uses an image pyramid to handle the different scale changes. Density maps can be iteratively refined to obtain higher-quality predictions. [13] proposes a two-stage approach where a low-resolution density map is refined into a high-resolution density map, while [14] uses feedback and [15] proposes a region-based refinement method.

Domain adaptation is essential for applying crowd counting technologies. [16] proposes to adapt a model to novel scenes by fine-tuning with a resampled dataset. [17] proposes a novel synthetic crowd dataset and a GAN-based algorithm to adapt from the synthetic dataset to real datasets. [18] proposes an adaptive convolution that uses the camera parameters as side information.

To exploit scene contextual information, [19] proposes a contextual pyramid convolutional network (CP-CNN) that uses global and local context to improve counting performance, while [20] uses the temporal context in videos. [21] proposes a ranking-based method to use unlabeled data in learning. A compositional loss is proposed in [22] to solve global counting, density map estimation and localization together. Other works have shown that density maps can be applied to improve the detection and tracking performance in crowd scenes [23, 24, 25]. A further survey of crowd counting algorithms can be found in [26].

Crowd counting algorithms are mainly concerned with counting the whole crowd, while in contrast fine-grained counting separates the crowd into groups and counts each group. While a crowd counting model could be learned for each group separately, this does not lead to good performance (see Sec. 5) due to low appearance variations between groups and ignoring the long-range context. In this paper, we propose a novel architecture for fine-grained counting consisting of two-branches, density map estimation and category segmentation, as well as density-based feature propagation to better exploit long-range contextual information, and complementary attention to mix information between branches.

Previous works on crowd analysis focus on classifying the crowd using high-level attributes, such as collectiveness, cohesiveness, and density level [27]. However, low-level individual behavior properties, such as facing direction, standing, sitting, are not considered. [28] proposes more properties of crowds such as stability, uniformity, and conflict which are mainly determined by the crowd motion. [29] proposes to classify crowd images by different crowd flow properties, while [30] proposes classifies whether a crowd image contains violent behavior or not. Detection of social interactions is considered in [31], but is only concerned with small groups (less than 4 people). Although crowd analysis algorithms can be used to distinguish different crowd types, they are mainly concerned with the high-level attributes of the crowd. In contrast to these works, the proposed fine-grained crowd counting aims to find the spatial distribution and underlying behavior properties of the crowd. Finally, general object detection [32, 33] also focuses on detection of individuals, but has difficulty distinguishing individual’s behavior since the inter-category appearance variation is low. Furthermore detection methods ignore the relationship and interaction between individuals in the crowd, thus are not able to capture the context information necessary for fine-grained counting.

To enable research on fine-grained crowd counting, we construct a new dataset with four fine-grained counting tasks, based on real-world requirements: 1) walking towards or away on a sidewalk; 2) standing or sitting; 3) waiting for bus or not waiting; 4) violent actions or non-violent. The images for the first task are from the UCSD crowd dataset [3], while the images for the fourth task are from a violence detection dataset [30] and a Kaggle dataset¹¹1https://www.kaggle.com/mohamedmustafa/real-life-violence-situations-dataset. We collected the images for tasks 2 and 3 from the Internet. To increase dataset diversity and evaluate algorithms under different image qualities, the dataset is composed of images with a variety of resolutions, use different color spaces (grayscale vs. RGB), and are captured from different sources.

Similar to crowd counting, the images were labeled with dot annotations indicating each person’s location in the image. In addition, each dot was also assigned a crowd category for fine-grained counting.²²2For the Towards/Away and Waiting/Not-waiting tasks, the raw video source was used for annotating the classes to reduce ambiguity. Since the tasks are orthogonal, we only label the classes specific for each task. Information about the dataset and typical images with annotations are shown in Table I and Figure 2.

To better understand the proposed dataset, we show the spatial probability maps of each class in Figure 4. We find that the spatial probability maps of each class are overlapping, making it impossible to distinguish different classes by a simple region-of-interest assignment map. Also, immediate adjacency to a point-of-interest (e.g., bus-stop) is not enough to predict the class, since behaviors can be defined by person-to-person context. For example, in Figure 2 (3rd column), the bus-stop is on the right, and the line context moves to the left away from the bus-stop).

We also visualize the average images for different tasks in Figure 5. For each task, the scenes are of similar context (e.g., street scenes), since they are task-specific. Towards/Away has a simple background due to the static camera. For other tasks, the background varies significantly.

We next discuss how to generate the ground-truth (GT) density and segmentation maps for training fine-grained crowd counting models. Let there be $K$ crowd categories indexed by $j\in\{1,\cdots,K\}$. Density maps are generated from a dot map by convolving with a Gaussian kernel. Specifically, given an image $I$ with width and height $(w,h)$ and the corresponding dot map annotation for the $j$th crowd category $D_{j}\in\mathbb{R}^{h\times w}$, the corresponding density map $Y_{j}\in\mathbb{R}^{h\times w}$ for the $j$th category is $Y_{j}=D_{j}*k_{\sigma}$, where $k_{\sigma}$ is a 2D Gaussian kernel with bandwidth $\sigma$ and $*$ is convolution. The kernel bandwidth can be fixed or change with pixel location, based on perspective [8] or crowdedness [8]. The generation of the density map is equivalent to placing a Gaussian on each dot annotation.

The GT category segmentation maps are generated from the GT density maps. First, the background segmentation map $S_{K+1}$ of low density regions is

where $\epsilon$ is a low-density threshold, and $\mathbbm{1}$ is the indicator function. The soft segmentation ground-truth for the $j$th category $S_{j}\in\mathbb{R}^{h\times w}$ is

where $\eta$ is a small number to prevent dividing by 0. Note that the GT segmentation maps are rough segmentations (not per-pixel GT as in semantic segmentation) since they are generated only from the density maps. For fine-grained crowd counting, this rough segmentation is sufficient since the main goal is to assign each non-zero density pixel to a category, rather than all the image pixels.

To encode contextual information for better segmentation of fine-grained classes, we propose a density-aware feature propagation method for the segmentation branch. The aim is to iteratively propagate features in the feature map along high density regions to encode context and generate refined maps. Note that semantic segmentation methods also use spatial smoothing of feature maps [35, 36]. However, in contrast to standard semantic segmentation, which applies the spatial smoothing uniformly, our density-aware propagation applies smoothing only on regions connected through high crowd density values, so that these context features do not affect people who are not connected in a crowd.

We consider three potential models for feature propagation: Conditional Random Fields (CRFs) [37], Graph Convolutional Network (GCN) [38], and Stacked Hourglass [39]:

• •

  CRF is a graphical model that is widely used in semantic segmentation to improve accuracy. The relationship between pixels is modeled with probabilistic inference under the assumption that similar pixels should have similar labels. Given an input image $I$ which has $n$ pixels, the segmentation of $I$ can be modeled as a random field $S=(S_{1},S_{2},\dots,S_{n})$ with conditional distribution

  $\displaystyle p(S|I)=\tfrac{1}{Z(I)}\exp(-E(S|I)),$

  (4)

  where the energy $E(S|I)$ is

  $$E(S|I)=\sum_{i}\psi_{u}(S_{i})+\sum_{i<j}\psi_{p}(S_{i},S_{j}).$$

  (5)

  $\psi_{u}(S_{i})$ is the unary potential that measures the negative log-likelihood of assigning pixel $i$ to segment $S_{i}$, and $\psi_{p}(S_{i},S_{j})$ is the pairwise potential of assigning pixels $i$ and $j$ to segments $S_{i}$ and $S_{j}$. Note that, message passing is performed in local regions and hand-crafted features are used for evaluating the similarity between pixels, as in [37].

• •

  GCN is a generalization of CNNs to graph structures, which are used to exploit contextual information in segmentation [40]. Given an image $I$ with $n$ pixels, the adjacency matrix between pixels $A\in\mathbb{R}^{n\times n}$ is defined by the cosine similarity between their corresponding deep features $F\in\mathbb{R}^{n\times d}$, where $d$ is the feature dimension. The feature propagation is defined as:

  $$H^{l+1}=\sigma(AH^{l}W^{l}),$$

  (6)

  where $H^{l}$ is the output of the $l$-th layer, $H^{0}=F$, and $\sigma(\cdot)$ is a non-linear activation, and $W^{l}$ is the weight matrix of the $l$-th layer. The propagation is performed in local regions.

• •

  Stacked Hourglass is an iterative bottom-up, top-down model that combines global and local contextual information. After the initial features are extracted, the features are iteratively refined using an hourglass module, as shown in Fig. 6. See [39] for more details.

Density-aware propagation: Since the context of neighboring people is important for crowd categorization (e.g., people waiting in line), we aim to encourage features to propagate mainly from person to person along high-density regions. Therefore, we propose a density-aware feature propagation method, where features propagating through low-density regions (background) decay faster than those in high-density regions (people). First, a dampening weight matrix $W_{d}\in\mathbb{R}^{h\times w}$ is defined based on the first-stage prediction of the density map $\hat{Y}^{\prime}\in\mathbb{R}^{h\times w}$,

where $\lambda\geq 0$ is the dampening factor and $\epsilon$ is a density threshold. Second, $W_{d}$ is used to decay features during propagation: for CRF, the message is multiplied by $W_{d}$ before being passed to other pixels; for GCN and Stacked Hourglass, the features are multiplied by $W_{d}$ before each GCN layer or hourglass module. In this way, features are propagated along high-density regions, and decay exponentially when moving through low-density regions.

To exploit the complementary information between the two branches, we propose a complementary attention model. First, the first-stage density map and segmentation map are generated. Next, the first-stage segmentation features and the first-stage density map are concatenated, and then fed into the refinement framework. The first-stage density map serves as the attention for the segmentation branch, which the model during feature propagation. Similarly, the refined segmentation map is concatenated with the first-stage counting features and serves as the attention for the counting refinement (see Fig. 3).

The output of the density map estimation branch is the overall density map for all people, which is multiplied by the category segmentation maps to obtain the fine-grained density maps. We use three loss functions during training: counting loss, segmentation loss and fine-grained loss. Counting loss measures the mean square error (MSE) of the overall density map with the predicted density map,

where $Y_{j}$ is the GT density map of the $j$th category, and $\hat{Y}^{\prime}$, $\hat{Y}$ are the first-stage (before the attention block) and final predicted density maps. Segmentation loss measures the category segmentation error via soft cross entropy (SCE),

where $S_{j}$ is the ground-truth segmentation map for the $j$th category, and $\hat{S}^{\prime}_{j}$, $\hat{S}_{j}$ are the first-stage and final predicted segmentation maps. Note that the segmentation maps are soft maps. Fine-grained loss is a category-specific loss that measures the fine-grained counting error for each category,

where $\hat{Y}_{j}$ is the predicted density map for the $j$th category. The final loss is computed by combining the three losses together,

where $\alpha$ and $\beta$ are hyper-parameters.

For the Towards/Away task, 800 images are used for training and 1200 images for testing following the train/test split of the UCSD dataset [3]. For the other 3 tasks, 100 images are used for testing and the remaining images used for training and validation. The learning rate is set to 0.0001 during training. For density-aware feature propagation, the dampening factor is $\lambda=0.2$, the density threshold is $\epsilon=0.001$, and 3 iterations of propagation are used, which were determined by an initial experiment (see Fig. 7). The hyperparameters of the final loss are $\alpha=100$ and $\beta=10$.

The count predictions using the fine-grained density maps are evaluated using mean absolute error (MAE), for each category $j$,

where $Y_{j}^{i}$, $\hat{Y}_{j}^{i}$ are the ground-truth and predicted density maps for the $i$th test image and the $j$th category, $n$ is the number of test images, and $\mathrm{sum}$ is the spatial sum over the map. The fine-grained counting performance is summarized with the category-averaged MAE (CMAE),

We also evaluate the predictions from each branch of the architecture. The count prediction of the overall density map is evaluated using overall MAE (OMAE),

The performance of the segmentation branch is evaluated using segmentation accuracy and recall of the non-background pixels. Note that we do not evaluate segmentation precision since the false-negatives will be ignored due to the corresponding low values in the density map. Furthermore, the accuracy and recall are only used to roughly estimate the segmentation performance, which is less important compared to CMAE since the segmentation ground-truth is not accurate.

We first conduct an ablation study to show the efficacy of our model components. The ablation study uses the Standing/Sitting task of the dataset.

Context: We first compare the three feature propagation methods with contextual information in Table III(a). The counting performance is improved when using feature propagation, compared to when no feature propagation is used. This confirms that the context is important to distinguish between fine-grained crowd categories. Comparing the three propagation methods, GCN does not perform well because the construction of the graph using feature similarity is not effective. CRF performs better than GCN but less effective than the Stacked Hourglass, most likely because we use location and pixel color as hand-crafted features for message passing. However, these features may be too similar for people in different categories. Finally, stacked hourglass can effectively capture the global and local context by learning, and achieves the beset performance.

To better understand the effectiveness of contextual information, we visualize the segmentation maps generated by different approaches in Fig. 8. Without contextual information (no feature propagation), the segmentation map is noisy as it is difficult to classify fine-grained categories by appearance features only. For GCN and CRF, the quality of the segmentation maps is improved. However, some areas are misclassified as circled in Fig. 8. The segmentation maps generated by Stacked Hourglass are most similar to the ground-truth.

Density-aware propagation: We next evaluate the effectiveness of using the density map to guide feature propagation. From Table III(b), our method using density-aware propagation (“Context+CoAtt+DAProp”) achieves better performance than “Context+CoAtt”, and “Context+DAProp” is better than “Context”, which confirms density-aware feature propagation is effective.

We also compare segmentation maps with or without DAProp in Fig. 9. Without DAProp, the CoAtt model is less confident about the classification of the white circled area because it will be affected by background pixels during propagation (white arrows). However, with DAProp, the model is more confident about the classification result since the background features decay faster in low-density area (i.e. background non-crowd).

Complementary attention: Next we investigate the complementary attention module (see Table III(b)). Our method (“Context+CoAtt+DAProp”) outperforms “Context+DAProp”, and “Context+CoAtt” is better than “Context”, which confirms the effectiveness of the complementary attention module. We also compare the proposed attention with naive attention (NaiveAtt), which directly multiplies the first-stage features with the attention map before refinement. Replacing CoAtt with NaiveAtt leads to worse OMAE performance, due to the loss of information after the NaiveAtt mechanism. This confirms the counting branch also benefits from the context information from the segmentation branch.

In Fig. 9, we compare the segmentation maps between methods with and without CoAtt. As circled in black, the human face is incorrectly segmented without CoAtt. In contrast, the same face is segmented more accurately when using CoAtt since the model can focus on high-density areas.

Loss functions: Finally, we evaluate different combinations of the loss functions: fine-grained loss (FineGrained), segmentation (Seg), and counting loss (Count). The results of various combinations are presented in Table III(c). Fine-grained loss is the most important loss function since the performance significantly decreases without it. The best performance is achieved by combining these three loss functions together. Finally, both Accuracy and Recall of the proposed method (“FineGrained+Seg+Count”) outperform “FineGrained+Count”, which shows that the segmentation loss is effective to improve the segmentation quality.

To confirm the effectiveness of our proposed method, we compare with four baseline methods:

OneNet: directly predict all the density maps for the categories with one network. Each category corresponds to one output channel of the network, and the feature maps are shared among the categories.

TwoNets: use a separate network for each category, i.e., no feature sharing between categories.

Segment: Use a shared feature extractor and predict an overall density map and a segmentation map (without using contextual information), which is equivalent to the first-stage prediction of our method. The category density maps are obtained by element-wise multiplying the density map with the segment map.

The results are presented in Table IV. We test different ground-truth density maps, generated via fixed bandwidth (4 and 16, denoted as Fix4 and Fix16), and adaptive bandwidth (Adaptive). Due to differences in image resolution between the tasks, Segment and Ours perform better using Fix16 on tasks with high resolution images (Standing/Sitting, Waiting/Not waiting), and Fix4 on tasks with low resolution images (Towards/Away, Violent/Non-violent). We present results using these settings for Ours.

The segmentation-based approaches (Ours, Segment) outperform direct prediction with one or two networks (OneNet, TwoNets), since the segmentation branch provides more meaningful supervision of the different categories, compared to a network learning a mix of density and category at the same time. Furthermore, our proposed method is consistently better than Segment (the first-stage prediction), which validates the usefulness of context information and information sharing between branches for solving the fine-grained counting task. Examples of fine-grained crowd counting results for different methods are shown in Fig. 10.

We also run an experiment with a stronger backbone – CSRNet which is pre-trained ImageNet [41]. Table V shows the results using CSRNet for fine-grained counting (either one network or two networks), and our method using CSRNet as the backbone: 1) with joint-training of the density map and segmentation networks (Ours); 2) only training the segmentation network (w/o joint train); 3) fine-tuning the CSRNet backbone and training the segmentation network separately (finetune w/o joint train). Comparing backbones for our method, the CMAE values are generally lower using the deeper pre-trained CSRNet than with the shallower FCN-7. Comparing methodologies, the results using the CSRNet backbone are consistent with those using the FCN-7 backbone – our method using context outperforms the simple one-net and two-net versions. Also, our method with joint training achieves better performance than methods without joint training, which confirms the effectiveness of training the segmentation and counting branches together.

We test two detection algorithms for fine-grained crowd counting: TinyFace [42] which detects human face and YOLOv3 [43] which detects human body. We fine-tune the pre-trained detectors to detect people from different categories. TinyFace is pre-trained on the WIDER Face dataset [44] and YOLOv3 is pre-trained on COCO [45]. Since bounding boxes are not labeled in our proposed dataset, we first generate ground-truth bounding boxes on dot annotations using the pre-trained detectors (TinyFace or YOLOv3) with a low threshold of 0.2. The class label is determined from the category density in the bounding box. Then, a detector for the crowd categories is fine-tuned on the fine-grained counting training set with the generated ground-truth bounding boxes. The category count is obtained by counting the number of detections for each category.

The experimental results using detector-based counting are shown in Table VI. TinyFace outperforms YOLOv3 on Standing/Sitting task because it is more reliable to detect faces than the whole human body in crowded scenes. For the other three tasks, which are less crowded than Standing/Sitting, the human body detector YOLOv3 is better than TinyFace. However, both detection methods are worse than our proposed approach for fine-grained counting. The detection approaches do not work well, compared to the density map approaches, because of the low appearance variations between crowd categories and need for long-range context.

In Table VII, we also compare the overall counting error (OMAE) of detectors and our approach. TinyFace achieves better overall counting performance than YOLOv3 on high density scenes, such as the Standing/Sitting and Waiting/Not-waiting tasks, because face detector works better for occluded scenes. For low density images from tasks like Violent/Non-violent, the YOLOv3 human body detector is better. The overall counting performance of our proposed method is significantly better than detection based approaches for all tasks.

We also visualize detection results of two detectors on the four fine-grained counting tasks in Figure 11. For crowded scenes like Standing/Sitting and Waiting/Not-waiting, TinyFace can detect most of the people in images, while the YOLOv3 body detector misses many people far away from camera or occluded by other people. For applications with less people, the YOLOv3 body detector is better than a face detector because more contextual information can be exploited. Both detectors achieve bad performance for fine-grained crowd counting because it is difficult to distinguish fine-grained categories only by appearance features. As shown in Figure 11, some of the detections are classified as both categories. This confirms that it is difficult to solve fine-grained crowd counting by detection based approaches.