Crowd Localization from Gaussian Mixture Scoped Knowledge and Scoped Teacher

As aforementioned, the counting community attacks intrinsic scale shift in two mainstreams. From the model perspective, a multitude of works [17, 18, 19, 20, 21, 22, 23] deal with intrinsic scale shift via multi-feature fusion. Moreover, some others [24, 2, 25, 26, 27] trace the essence of intrinsic scale shift namely perspective imaging, which utilizes the predicted perspective map as training strategy. Despite that perspective related works achieve certain promotion, we assert the intrinsic scale shift has not been aligned. To this end, Auto Scale [28] and L2SM [29] propose to scale the image patches according to density level. [30, 31, 32] also feed patches with distinct density level into CNN with different receptive fields. [7] presents the crowd flow to enhance counting performance with location flow. However, the density level cannot represent instance scale. Therefore, SD Net [16] introduces instance scale and use it to align scale shift. But SD Net fails to keep semantic information during handle and ignores intrinsic scale shift within the divided patches.

Crowd localization aims to locate the precise position of each head shown in the image. The very first idea about the localization must be object detection [33, 34, 35]. TinyFaces [36] utilizes a detection based framework via the analysis to the impacts of scales, context semantic information and image resolution to locate the tiny faces. Following TinyFaces [36], some researchers [15, 37, 38, 39] make extending work in tackling intrinsic scale shift. However, due to the shortage of detection structure, the detection based methods still perform poorly under extremely congested scenarios. Thus, some researchers begin to utilize regression based crowd locator. RD Net [40] leverages depth information to generate spatial aware supervision map. But in mainstream datasets of crowd localization, the depth information is unavailable. Thus, [6] proposes to utilize fine-grained density map to make crowd localization. BL [12] proposes a location aware loss function to locate the crowds. But it fails to address intrinsic scale shift. [41, 42, 43, 44, 45] utilize instance segmentation to locate crowd. Especially, the instance segmentation locators introduce box annotation in regression. By this way, the instance scale information can be estimated. Thus, our method follows this baseline. What’ s more, there are still some other works concentrating on intrinsic scale shift of crowd localization. Auto Scale [28] proposes to estimate a density region and learn to zoom it. Similarly, RAZ Net [10] also proposes a selection strategy to select the density region. These zooming strategies cannot cater to multi-region variance.

The original proposal of the teacher-student model serves transfer learning. [46, 47, 48, 49] utilize teacher-student model in Knowledge Distillation (KD). Actually, our Scoped Teacher model is inspired by KD, in which the teacher model plays a role in bridging data with student model. However, the teacher model in KD tends to utilize a larger teacher model to exploit latent knowledge. However, our Scoped Teacher model shares the same architecture with student model and the images fed into teacher model have been processed by GMS, in which the latent knowledge is not from model representation capacity but from GMS. In Semi-Supervised Learning (SSL), some researchers also introduce teacher model. [50, 51, 52, 53] introduce teacher model in doing Consistency Regularization. In [54], they introduce a momentum network to predict pseudo label for unannotated images, which is actually a teacher-student model. The teacher models used in SSL are inclined to predict pseudo labels for unannotated samples which tend to be coarse knowledge. Despite that the proposed teacher model also aims to use Consistency Regularization, our target is to transfer fine-grained knowledge not coarse knowledge which has been learned by student crowd locator with annotation.

The popular density map regression method in crowd counting cannot provide precise spatial information. Therefore, some researchers [41, 42] introduce to segment instance-head to make crowd localization. Concretely, they utilize a fixed and global threshold to transfer regressed confidence map activated by a sigmoid function into binary map, which is not robust. Therefore, IIM [43] proposes an additional and trainable pixel level threshold map to binarize confidence map.

Formally, given an image $\mathcal{I}_{ori}\in\mathbb{R}^{3\times H\times W}$, in which the footnote $ori$ represents the original resolution images, a confidence map $\mathcal{F}_{ori}\in\mathbb{R}^{1\times H\times W}$ is predicted, see Eq. 1,

where $\mathbf{0}$ and $\mathbf{1}$ denote the tensor filled with $0/1$. Additionally, for the fixed threshold works, the segmented binary map $\mathcal{B}_{ori}^{fix}\in\mathbb{R}^{1\times H\times W}$ is obtained through Eq. (2):

where $\varepsilon$ is a fixed threshold and $v,h$ are the pixel coordinates. As for the IIM, the binary map $\mathcal{B}_{ori}^{apt}\in\mathbb{R}^{1\times H\times W}$ is obtained through a trainable threshold map $\mathcal{T}\in\mathbb{R}^{1\times H\times W}$ as Eq. (3):

In Eq. 3, it is obvious that the process is non-differentiable. Thus, [43] proposes to relax it to provide a gradient which can be described as Eq. 4:

where $\alpha$ is the learning rate and $\mathcal{L}_{seg}$ is formulated as Eq. 6.

With the adaptative threshold map $\mathcal{T}$, a robust binary map $\mathcal{B}_{ori}^{apt}$ is derived. The training strategy is formulated as Eq. (6):

where $\mathcal{\widehat{B}}\in\mathbb{R}^{1\times H\times W}$ is the ground-truth binary map of image $\mathcal{I}_{ori}$. By this way, a precise binary map is derived. Therefore, we follow the mentality of IIM as our baseline work. To clarify the paper, we omit the $apt$ in $\mathcal{B}_{ori}^{apt}$ in the following.

In instance segmentation crowd localization, since the locator derives the supervision signal from binary map with implicit scale information, in which the head-areas are annotated as the foreground, the locator is fragile and sensitive to instance scale shift. Moreover, the performance of locator is limited for being hard to catch large and tiny instances simultaneously. We assert that the issue can be blamed on the chaotically distributed scales. In the light of deep model trained via Empirical Risk Minimization (ERM), it is arduous for crowd locator to converge on data not satisfing with independent identically distributed conditional assumptions. Summarizing the above analysis, the regularization for the chaotic scale distribution can be the point to tackle the intrinsic scale shift.

To this end, we propose to decouple the chaotic scale distribution into several regular sub-distributions. Therefore, the chaos within the scale distribution is transferred into the distribution shift among sub-distributions. Additionally, we constrain the spatial feature to be correlated with scale distribution in decoupling. By this way, the sub-distributions are compact in spatial features, and the sub-distributions alignment is available to be implemented via image interpolation.

Specifically and formally, given an image $\mathcal{I}_{ori}$ with $N$ pedestrians, in which the footnote $ori$ represents the image in the original resolution, and the corresponding scale distribution $\mathcal{S}_{ori}$ is shown as Eq. (7):

where $\alpha_{i}$ is the scale for $i^{th}$ instance and $\delta$ denotes one-dimension Dirac function. We utilize a Gaussian mixture distribution to adapt to the $\mathcal{S}_{ori}$ as Eq. (8):

where the mixture distribution is composed of $c$ sub-Gaussian distributions $\mathcal{N}(\cdot)$ with parameters $\theta$ which are mean and variance in Gaussian distribution, and probability $\pi_{c}$. The $v$ is vertical location to the instance. The mixture model is initialized and updated from the Expectation Maximization, which has an objective function of:

in which $\Theta$ is the set of $\{\pi_{c},\theta_{c}|c=1,...,C\}$ and $\lambda$ is defined as Eq. 10:

As aforementioned, to facilitate alignment, the mixture model should be correlated with spatial feature in decoupling. Therefore, we only adopt the vertically spatial feature $v$ to reduce the computational complexity from $\mathcal{O}(n^{2})$ to $\mathcal{O}(n)$. As for the horizontal ones, we demonstrate the redundance of it in scale feature representation, see Section IV-C5. Practically, the fine-grained scale information is unavailable in all existing datasets. Therefore, the annotated box area $\alpha$ is adopted as the observation value to the mixed distribution.

After decoupling the mixture distribution, the chaotic scale distribution is decomposed into $c$ normal distributions, which seems to be feasible for model to converge. What’ s more, constraining the vertical features in adaptation and decoupling, the sub-distributions are compact spatially. Hence, $c$ patches are derived, where each one has a scale distribution of normal sub-distribution in Eq. (8). To this end, the issue in intrinsic scale shift is to align the scale shift among sub-distributions. Thus, we introduce some prior knowledge, in which an optimal scale $\alpha_{0}$ is set as the landmark to align the scale shift among sub-distributions. For each patch $p_{c}$, the aligned one $\widehat{p}_{c}$ is derived via Eq. (11) with an interpolation:

where $N_{c}$ is the count of instance in $p_{c}$. Note that to avoid computational cost, we make compromise on using average scale of patches. Since the decoupling provides the compactness of scale within the sub-distribution namely the patch, the average scale is adequate to represent the patch.

Finally, the scale shift is aligned in two levels which are inter-patch and intra-patch. However, the sub-distributions are still discrete. There are two kinds of normal process, in which one is to directly splice them and make padding on smaller ones, while the other is to keep them being discrete. Nevertheless, in sub-distributions alignment, the patches are interpolated via distinct scale factors, it is unavoidable for the junction region being distorted semantically. Moreover, since the decoupling is adaptive, the decision boundary is uncertain, which incurs that some to be detected instances could be cut off and distorted. To alleviate the issue, we propose a sub-distribution re-aggregation trick.

Re-Aggregation for Sub-Distributions. In aforementioned two processes to discrete sub-distributions, the uncertainty of decision boundary for decoupling incurs the risk for instances being cut off. Given an image or patch with instances cut off, the locator cannot catch the semantic and detect them. Thus, we argue that it is necessary for locator to be fed with whole image. As the Fig. 3 shown, let $\mathcal{I}_{scp}$ be the re-aggregated image. To meet the argument, the Eq. (12) must hold:

Therefore, given $c$ patches with $c$ counts corresponding scale factors $\left\{\alpha_{1},\alpha_{2},\cdots,\alpha_{c}\right\}$ which are from Eq. (11), we interpolate the $\mathcal{I}_{ori}$ with $\alpha_{i}$ for $c$ times and derive Eq. (13):

Then, the $\left\{\mathcal{I}_{scp}^{i}\mid i=0,1,\cdots,c\right\}$ are fed into locator which makes it catch correctly semantic information. To obtain the final prediction $\mathcal{B}^{re}_{scp}$ with spatially semantic mapping relation to original image $\mathcal{I}_{ori}$, the predicted $\mathcal{B}^{i}_{scp}$ from $\mathcal{I}_{scp}^{i}$ are re-interpolated via $\left\{\frac{1}{\alpha_{1}},\frac{1}{\alpha_{2}},\cdots,\frac{1}{\alpha_{c}}\right\}$ and the ROI is shot according to $p_{c}$. The finaly prediction is composed of shot ROIs.

In Section III-B, the Gaussian Mixture Scope (GMS) is proposed to regularize the chaotic scale distribution. Given a set of images $\mathcal{I}$ with chaotic scale distribution, the crowd locator represents $\mathcal{I}$ into latent space, in which the partial instances with certain scales are caught and partial backgrounds are mis-embedded. The GMS aligns scale facilitating feature embeddings with variant scale representation to be mapped to the same latent space. We name the predicted knowledge as $\mathcal{K}_{gt}$ which denotes that they come from ground truth trained model via ERM. The GMS processes $\mathcal{I}$, in which the outliers not in $\mathcal{K}_{gt}$ are represented to the same latent space and conform $\mathcal{K}_{GMS}$. In training phase, GMS exploits the latent $\mathcal{K}_{GMS}$ to make crowd locator catch the outliers better. However, despite that the implementation of GMS provides additional $\mathcal{K}_{GMS}$ to crowd locator, it is not the active learning, but the passive reception for relationship between $\mathcal{K}_{GMS}$ and locator. As a result, the training phase provides annotation for GMS to exploit $\mathcal{K}_{GMS}$ and aid locator to perform better, while the annotations are agnostic in testing phase and there is no $\mathcal{K}_{GMS}$ which incurs the locator perform poorly with only representation capacity for $\mathcal{K}_{gt}$. What’ s more in backpropagation phase, GMS dose not have a gradient to compute and its process is non-differentiable. Thus, there should be another better way to deploy GMS and make locator actively learn the $\mathcal{K}_{GMS}$.

To transfer the exploited $\mathcal{K}_{GMS}$, we propose a Scoped Teacher which is a teacher-student framework. Specifically, given an $\mathcal{I}_{ori}$, the student locator has a prediction of $\mathcal{F}_{ori}$ and $\mathcal{B}_{ori}$ which are confidence map and binary map. As for teacher end, the GMS is adopted to regularize the $\mathcal{I}_{ori}$. Then, the processed $\mathcal{I}_{scp}$ is fed into teacher locator to aggregate $\mathcal{K}_{GMS}$ and $\mathcal{K}_{gt}$. The $\mathcal{B}_{scp}^{re}$ is from further proceeding of sub-distribution re-aggregation. Then, a consistency loss is introduced as Eq. (14) to transfer $\mathcal{K}_{gt}$ from Scoped Teacher to student locator.

In consistency regularization, the Scoped Teacher adopts GMS to exploit $\mathcal{K}_{GMS}$ and restore it in the representation of $\mathcal{B}_{scp}^{re}$. With Eq. 14, the consistency constraint makes the $\mathcal{F}_{ori}$ and $\mathcal{B}_{ori}$ be closer to $\mathcal{B}_{scp}^{re}$. By this way, during training, the back propagation of consistency loss pushes the $\mathcal{K}_{GMS}$ being transferred from Scoped Teacher to student model.

Comparing with ground truth supervision, the improvement Scoped Teacher generated is more than GMS exploited knowledge $\mathcal{K}_{GMS}$ transform. In settings of Scoped Teacher, we leverage a shared architecture with student locator. Empirically, to utilize a larger model in teacher end designing could make teacher with stronger representation capacity guide weaker student training. However, our Scoped Teacher shares the same architecture to student model, which means the outputs between student and teacher ends are with more consistent. Therefore, the guidance of Scoped Teacher to student is feasible to implement consistency regularization.

Finally, in parameters updating, the student crowd locator is trained via gradient descend. To aggregate the knowledge and stable knowledge transform, the teacher parameters $\vartheta_{t}$ is updated via Exponential Moving Average (EMA) with student parameters $\vartheta_{s}$ as Eq (15):

where $m$ denotes the EMA decay coefficient to control the updating rate.

Instance Segmentation Loss. The instance segmentation loss is a L2 loss for confidence map regularization and a L1 loss for binary map regularization as Eq. (6).

Consistency Regularization Loss. Since the gradient is detached in threshold learner, to optimize the threshold learning, L1 loss is used for binary map regularization between teacher and student end as Eq. (14).

Total Loss. During training, the student model is jointly trained in an end-to-end manner. The whole parameters are updated by integrating all mentioned loss functions:

The teacher model is optimized as Eq. (15).

At the testing or inference phase, the original image $\mathcal{I}_{ori}$ is fed into student model, which can be described as Eq. 17:

Therefore, our proposed method would not incur any extra costs in inference.

• 1

  Shanghai Tech Part A (SHHA): SHHA [55] contains 482 images where 270 are available for training, 30 are for validation and others are for test. There are 241, 677 instances annotated in total.

• 2

  Shanghai Tech Part B (SHHB): SHHB [55] consists of 716 images where 360 are prepared for training, 40 are for validation and others are for test. SHHB has 88, 488 annotated instances.

• 3

  NWPU-Crowd (NWPU): NWPU [11] is the largest dataset in crowd analysis community so far, in which there contains 5,109 images with 3,109 of them for training, 500 for validation and 1,500 for test.

• 4

  UCF-QNRF (QNRF): QNRF [9] is a dataset with extremely congested scenarios, where it is composed of 1,535 images and 961 of them are for training, 240 are for validation and others are for test.

In the training phase, backbone networks of VGG-16 [56] and HR-Net [57], a batch size of 6, an optimizer of Adam [58] with learning rates 1e-5 for backbone and 1e-6 for threshold encoder, a learning rate decay of 0.99 for every epoch are adopted, an interpolation mode of Bilinear. In the testing phase, the tested images are fed into student locator in original scale and the model with best performance on validation set is picked for testing. Moreover, our experiments are applied on two NVIDIA RTX 3090 with a total memory of 48 GB.

Following [43, 11], the Precision (Pre.) , Recall (Rec.) and F1-measure (F1-m) are adopted for localization metrics as Eq. (18),

where F1-m is the core norm and $TP,TN,FP,FN$ denote True Positive, True Negative, False Positive and False Netgative. The MAE, MSE and NAE are adopted for counting metrics as Eq. (19),

In this section, four chosen datasets are grouped into three parts. NWPU and JHU are comprehensive dataset where the congested and sparse scenarios are all included. QNRF and SHHA are congested datasets, while SHHB is the sparse dataset.

In this section, we discuss how GMS and Scoped Teacher improves the final performance based on experiments in Section IV. To facilitate clear discussion, we pick one typical sample from SHHA and visualize its predicted confidence maps, threshold maps and binary maps from models of baseline method, GMS inferred and Scoped Teacher transferred, as shown in Fig. 6.

To begin with Tab. I, we notice that directly implementing GMS in the testing time brings improvement. Thus, it demonstrates the effect of scale alignment. However, it goes as a common sense that the conventional image interpolation would not provide any additional information. To this end, we argue that the improvement by GMS comes from distribution regularization and other forms of knowledge exploitation. See the column of Confidence in Fig. 6, the red box selects a region filled with tiny instances. In the bottom of box, the GMS provides higher confidence than baseline. This is because the original representation of those instances with improved confidence is still in the latent space of crowd locator. Hence, this is the GMS exploited knowledge. Nevertheless, since there is only an effective resolution (dataset author provided resolution) of 1024 * 768 to the image in Fig. 6, the instances in the top of the red box have representations of outliers, which are still outliers after alignment via GMS. With confidence variance by GMS explained, we put concentration on threshold learning. See the black box in the Threshold column of Fig. 6, GMS brings unsmooth distribution to threshold map. Actually, the regularization of GMS does not introduce any influence on model parameters. In the right and left side of the black box, the two regions with obviously low thresholds should be negative. It is blamed on the poor robustness of the baseline model. Since the corresponding area in red box shows better prediction on confidence, the abnormal low threshold area in black box is hard to be explained by non-semantic distortion. In the Scoped Teacher training, GMS exploits the wrong prediction actively to teacher model to correct them, see black box in the Scope row of Fig. 6. Thus, this is also the GMS exploited knowledge.

Then, we discuss the effect of the Scoped Teacher. There may be interests on how the knowledge transfers and what role of the Scoped Teacher plays beyond bridge in transform. Similarly, beginning from confidence analysis namely Confidence column in Fig. 6, the red boxes select our ROI. We assert that Scoped Teacher guides student to build a connection between normal tiny representations with GMS aligned representations. This is because the student locator fed with original outliers is inclined to make prediction being similar with teacher fed with mapped embeddings and the process makes implicit mapping transform from outliers embedding to normal embedding. What’ s more, there is another interesting phenomenon that the confidences are higher in red boxes when comparing Scope column with GMS column. We argue that the Scoped Teacher makes a further aggregation to the knowledge. The scope of GMS is limited within the temporary input. However, the Scoped Teacher learns to build the connection from all similar representation in the training set. Thus, the knowledge in confidence is the connection and the role Scoped Teacher plays is the connector and aggregator. Then, we analyze the Threshold learning. See Threshold column in Fig. 6, the black boxes also select our ROI. We notice that the threshold from Scope is more smooth than GMS. In consistency regularization, the negative regions outputs FP samples which exposes the un-robustness of locator and the corresponding loss is deployed to optimize the un-robustness. What’ s more, since GMS treats distribution discretely, it makes images lost physical features. The Scoped Teacher aids locator learn GMS exploited useful information and ignore the issues incurred by losing physical feature. Thus, the knowledge transferred is the punishment on un-robustness and the role Scoped Teacher plays is a filter to select useful knowledge. Finally, we put analysis on Binary column of Fig. 6. See the yellow boxes, the Scope depicts less boxes than GMS. And the recall comparison shows there are more boxes are removed, while the precision comparison shows there are more accurate boxes predicted. Thus, the knowledge transferred is box refinement and the role Scoped Teacher plays is the refiner.