LLA: Loss-aware Label Assignment for Dense Pedestrian Detection

Recently, occlusion handling becomes a popular topic in the field of pedestrian detection. In heavy occlusion situations, the detector will get confused by the adjacent instance which leads to an inaccurate boundaries regression. To solve this problem, Repulsion Loss [27] and OR-CNN [33] both impose additional penalty terms on the BBoxes that appear in the middle of two persons to force them to regress to the right person. Moreover, utilizing visible annotation as extra supervision is a common strategy to obtain more precision location information. Bi-box [37] adds a visible branch on Fast R-CNN to predict the full and visible body of a pedestrian at the same time. ATT [34] exploits the visible-region information as external guidance to handle various occlusion patterns in crowded situations. MGAN [18] forces the detector to focus on the visible regions of a pedestrian by adopting a novel attention branch to highlight the visible body region while suppressing the occluded part. What’s more, some researchers point out that in crowded scenario NMS may be trapped in a dilemma: a lower threshold of intersection over union (IoU) resulting in the miss of highly overlapped pedestrians while a higher IoU threshold naturally brings in more false positives. To solve this problem, Adaptive-NMS [16] proposes a subnet to predict the threshold for different anchors. R2-NMS [8] leverages the less occluded visible parts to remove the redundant boxes. PS-RCNN [5] utilizes two parallel R-CNN modules to detect slightly/none occluded and heavily occluded human instances in a divide-and-conquer manner. Different from all the existing works, our method handles the severe occlusion situation by a novel label assigning method, which neither requires additional annotation nor demands extra parameters.

Anchors are a set of pre-defined square boxes with different scale and aspect ratios which are densely assigned to each spatial location on feature maps. Traditional anchor-based object detectors [14, 20, 13, 1] assign labels for anchors based on hand-crafted IoU between anchors and GTs. Specifically, if an anchor’s IoU with a certain GT box exceeds a certain threshold (e.g. 0.5 for RetinaNet [14]), this anchor is defined as positive. The remaining anchors are either defined as negative or ignore according to its maximum IoU between GT boxes. Besides, extra scale constraint introduced by FPN [13] is imposed into these detectors [13, 1] to better handle the scale variations in objects. Instead of using pre-defined anchors, MetaAnchor [29] proposes an anchor function to provide anchors with dynamic shapes in both training and testing phase. Guided Anchoring [25] first learns a set of adjusted anchors in an anchor-free manner to better fit the shape of targets, and then design the subsequent anchor-based modules based on the learned anchors.

Anchor-free methods have drawn more and more attention recently due to its simple pipeline. FCOS [24] and FoveaBox [10] define anchors in the center region of targets as positive anchors. FSAF [41] further utilizes an online Feature Selection Module to select the appropriate feature level for each GT. In keypoint-based anchor-free detectors [11, 39, 38, 30], only a single center point for each GT (anchor which is the closest to the center of that GT) is defined as positive, while other anchors are all negatives. Because the mechanism of keypoint-based detectors is much different from bounding box based detectors, they are out of the scope of this paper.

Recently, methods called dynamic label assignment are proposed to improve the training process of detectors. ATSS[32] introduces dynamic IoU thresholds by calculating the mean and standard deviation of IoU values on a set of pre-defined candidate anchors for each GT. FreeAnchor [35] designs a detection-customized likelihood that takes precision and recall into consideration to tackle the anchor-object matching problem. Specifically, anchors that have a higher likelihood are defined as positives. PAA [9] first designs a score function based on the classification and regression loss then applies one-dimensional GMM [21] for these calculated scores for each GT to choose the thresholds for separating positive and negative anchors. DeTR [3] and DeFCN [26] explore customized loss and quality terms as their indicators for one-to-one label assignment. Although one-to-one matching strategy is proved crucial to end-to-end detectors, it may not be the optimal choice for detectors followed by NMS.

Instead of solving the anchor-object matching problem directly, there are also a few works trying to re-weight the positives and negatives which can be categorized into generalized label assignment. PISA [2] first proposes two ranking strategies – IoU-HLR and Score-HLR to rank positive and negative proposals, respectively, to evaluate the importance of anchors, then forces the model focus on more important samples by giving them higher weights. Noisy Anchor [12] constructs a cleanliness score based on the detector’s outputs to re-weight anchors and soften classification labels. However, as we stated above, although they can adaptively define positive/negative anchors, they still limit the positive regions near the geometric center of each GT box, which could harm the detectors’ performance in DPD.

Let us take a brief look at how label assignment is conducted on two well-known one-stage detectors – RetinaNet and FCOS. Given an input image $M$, the ground-truth annotations are denoted as $G$, where a ground-truth box $g_{i}\in G$ is made up of a class label $g_{i}^{cls}$ and a location $g_{i}^{loc}$. Due to the wide-spread of multi-scale feature pyramids network (FPN), both scale and spatial constraints need to be considered during label assignment.

In RetinaNet, $a_{j}\in A$ stands for an anchor box. RetinaNet handles spatial and scale constraint simultaneously based on the Intersection-over-Union (IoU) matching rule. During training, $a_{j}$ is assigned to GT $g_{i}$ if $IoU(a_{j},g_{i}^{loc})>0.5$, while $a_{j}$ is defined as negative if ${\forall}g_{i}\in G,IoU(a_{j},b_{i}^{loc})<0.4$. Anchors which are neither positive nor negative are ignored at that training step.

In FCOS, $a_{j}\in A$ stands for an anchor point. During training, $a_{j}$ is assigned to GT $g_{i}$ only if 1)$a_{j}$ falls into the center area (within a fixed radius) of $g_{i}$. 2) $a_{j}$ meets the pre-defined scale constraint introduced by FPN[13]. Anchor points which do not meet these two requirements are defined as negatives. Note that in FCOS, both spatial and scale constraints are explicitly imposed in the process of label assignment, making it less flexible.

Label assigning strategies in RetinaNet and FCOS are based on a strong assumption that the geometric center is the most appropriate spatial location to represent objects. In statistics, this assumption may hold when we deal with objects with a large variety of categories. However, compared to general object detection, dense pedestrian detection are different in the following two aspects:

• •

  Poses between different individuals vary a lot.

• •

  Human body can be heavily occluded by others.

Given these two specificities, heuristic label assigning strategies adopted in RetinaNet or FCOS are no more suitable for dense pedestrian detection. In Fig. 1, we visualize some typical scenarios in which inappropriate label assignment is introduced by RetinaNet and FCOS. As can be seen in Fig. 1, when the human body exhibits unusual postures (i.e. dancing, playing sports, etc), part of positive anchors of both RetinaNet and FCOS may fall onto background regions. In these cases, the classifier will get confused and then learns improper decision boundaries. When people are very close to each other, their positive anchors will be severely intertwined where the regressor can hardly determine which target should be approached for each anchor. These examples indicate that in dense pedestrian detection, hand-crafted label assigning strategies are no more qualified.

Some recent works [32, 9, 35] have taken advantage of the outputs of models to perform label assignment, referred to as dynamic label assignment. The success of dynamic label assignment is based on an observation that anchors with lower loss/higher likelihood can better represent the corresponding instances. However, all these strategies need to first construct a positive candidate set of anchors for each object and then adaptively split positive and negative anchors according to the statistics from the candidate set. As the construction of the candidate set remains hand-crafted, the whole set is still near the geometric centers of their corresponding objects, which makes them partially-dynamic. As shown in Fig. 1, in ATSS, some of the positive anchors of a heavily occluded human instance, still fall into other instance’s body region, which indicates that partially-dynamic label assignments are sub-optimal solutions for dense pedestrian detection.

Based on previous discussion, we propose Loss-aware Label Assignment (LLA), a simple and fully-dynamic label assigning strategy for dense pedestrian detection which can be easily plugged into any anchor-based or anchor-free detectors. Without loss of generality, we extend the definition of $a_{j}\in A$ into an anchor box/point. Given an input image M, suppose there are $J$ anchors and $I$ GT annotations. In a single forward step, we can get score predictions $S(\theta,M)\in\mathbb{R}^{J\times N}$, where $N$ is the number of classes and $\theta$ is the learnable parameters in detector, and get bounding box predictions $B(\theta,M)\in\mathbb{R}^{J\times 4}$. Unlike previous methods which only calculate losses between each anchor and its assigned GT, LLA calculates losses between all anchor-gt pairs, getting:

In our implementation, the term $+\infty$ is replaced by a large positive value (e.g. $10^{2}$). Then, the Restricted Cost Matrix can be formulated as:

After selecting the top $K$ smallest values in $C$, finally, the assignment matrix $\pi_{ij}\in\{0,1\}$ can be obtained:

Note that if an anchor is assigned to multiple GTs simultaneously, we assign this anchor to GT with the smallest cost. After label assigning, the detector is updates in the same as in RetinaNet and FCOS.

Compared to label assigning strategies with complex hand-crafted rules, LLA only leverages the minimal “in box” prior. Which FPN layer should each GT assigned to is automatically determined by LLA according to the feedback of the model’s outputs. Compared to partially-dynamic strategies that restrict the positive candidates near the center of each GT, LLA assigns anchors in a fully dynamic manner which helps LLA better handle the severe occlusion situations.

For CrowdHuman, the network structure in our experiment follows [14]. We resize the input images so that the short edge is 800 pixels while the long edge is smaller than 1400 pixels. We train our model on 8 GPUs with 16 images per mini-batch. SGD with the momentum of 0.9 is adopted as our optimizer. The initial learning rate is 0.01 for both RetinaNet and FCOS and is decayed by a factor of 10 after 8th epoch and 11th epoch. The training process finishes at the end of the 12th epoch. If not specified, the default backbone in our experiments is ResNet-50 [7]. We adopt IoU Loss for $f^{reg}$ in Eq. 1. For back-propagation, the regression loss is replaced by GIoU Loss because we find such a usage of IoU/GIoU Loss can yield the best performances. For evaluation, we follow the standard Caltech [4] evaluation metric – MR, which stands for the Log-Average Missing Rate over false positives per image (FPPI) ranging in $[10^{-2},100]$.

Visualizations of positive anchors assigned by LLA’s are shown in Fig. 2. As seen in the first column, instead of evenly distributed in each GT box, positive anchors for these dancers fall onto the foreground more compactly. In the second and third columns, positive anchors of heavily occluded human instances fall onto the visible regions (e.g. heads, shoulders, etc.) which are far away from the geometric center of GT boxes, demonstrating the effectiveness of LLA.

A lower AAR means the adopted label assigning strategy assigns positive anchors in a more deterministic way, which is a desired property in the crowd scenario. We calculate the AAR for RetinaNet and FCOS when w/ and w/o LLA on CrowdHuman. Results shown in Table 5 illustrate that our proposed LLA can effectively decrease the AAR and reduce the ambiguity introduced in label assignment.

Note that LLA can be compatible with recent advances in DPD, thus in this section, we only compare LLA with other state-of-the-art label assigning strategies. NMS thresholds 0.5 and 0.6 are adopted for anchor-based and anchor-free methods, respectively. As seen in Table 6, For the anchor-based method, our LLA built upon RetinaNet surpasses all other methods without any bells and whistles (e.g. GroupNorm [28] and IoU Branch). For anchor-free methods, LLA is built upon FCOS. It needs to be mentioned that both FCOS and ATSS restrict positive anchors in the central region of an object, and thus they can benefit from the Centerness branch to eliminate false positives. Especially under MR – a metric which is extremely sensitive to false positives, using Centerness can remarkably reduce MR for FCOS and ATSS. However, LLA does not acknowledge and use center prior in crowd scenarios, under this circumstance, adopting the Centerness branch will instead hurt the detector’s performance. For fair comparison, we adopt the IoU branch proposed in PAA as a replacement of Centerness. As shown in Table 6, LLA without IoU branch leads FCOS and ATSS w/o Centerness by more than 5% MR, also shows better results than ATSS w/ Centerness. After further utilizing IoU branch, LLA surpasses ATSS by a clear margin.

We compare the prediction results with and without LLA in Fig. 5. The first two columns exhibit that LLA can successfully detect the mis-detected instances by original RetinaNet. The last two columns show that LLA can effectively reduce false positives. We believe such two merits in LLA mainly benefit from the better placement of positive anchors, which greatly reduces the ambiguity during the training stage.

CityPersons [31] is another dataset for pedestrian detection which consists of 2975 images for training, 500 and 1575 images for validation and testing. Following the evaluation protocol in CityPersons, objects whose heights are less 50 are ignored. Besides, the validation set is further divided into two subsets according to visibility – Reasonable and Heavy Occlusion. MR on these two subsets is reported in this work. We follow the same training details as in CrowdHuman. IoU branch is adopted in Anchor Free LLA. As shown in Table 7, our anchor-based and anchor-free LLA reduce MR on the Heavy Occlusion subset by 2.22% and 3.10% respectively, exceeding all other existing label assigning strategies.