Being Aware of Localization Accuracy By Generating Predicted-IoU-Guided Quality Scores

Compared with two stage detectors [31, 32, 33, 34], one-stage counterparts [13, 20, 21, 30, 35] own faster inference speed but lower accuracy because of directly predicting classification confidence scores and localizations, where occurs a false positive problem [5, 10, 15, 30, 31]. To solve this problem, Consistent Optimization [9] focuses on matching the training hypotheses and the inference quality by utilizing of the refined anchors during training. GA [34] leverages semantic features and adapts features based on predicted anchors’ scales and aspect ratios at different locations. RefineDet [26, 27] propose a novel detection architecture, where the object detection module(ODM) further refines positive anchors predicted by anchor refine module(ARM) based on features aligned by transfer connection block(TCB). FoveaBox [30] adopts DCN based on learned box offsets to improve its performance. AlignDet [36] combines the flexibility of learned anchors and the preciseness of aligned features. As CLQ belongs to one-stage detector series, in this paper we also follow a design of feature alignment branch for CLQ based on predicted anchors and DCN to improve the performance.

As classification confidence score is usually be treated as a reflection of predictions’ qualities [1, 2, 3, 6, 13], making some false positives [5,10] with high classification scores but low localization quality be selected during inference, which inevitably degrades the performance. To alleviate this problem, forked with some feature selection models [5, 10, 13, 30, 37] aiming to select better positive samples across and within FPN feature levels, LQE-based methods aim to enable detection process to aware both classification and localization accuracy. IoU-Net [1] and IoU-aware [4] learn to predict the IoU between each detected bounding box and the matched ground-truth to acquire the score of localization accuracy, thus well localized bounding boxes are preserved in the NMS procedure. FCOS [13] and ATSS [14] proposed “Centerness” to lower samples’ contributions that are far away from center of positive regions, treating distance a reflection of localization quality. GFLV2 [3] performs LQE using Distribution-Guided-Quality-Prediction module based on the learned distributions of the four parameters of the bounding box. Beyond the independent design of LQE branch that may also encounter an inconsistent of classification and localization quality [2], some other works modulate classification loss by adding samples’ IoUs as weight factors such as Consistent Loss [8], Noisy Anchors [6]. More directly, some soft-label-based methods [2, 3, 6, 24] change the classification scores’ targets by a function of classification score and IoU during training. Inspired by all these excellent works, for well making consistent of classification and localization-quality, and taking a ride of the close relationship between LQE and IoU, we propose the CLQ detector where a novel joint classification-localization-quality representation is designed through combining classification and predicted-IoU-guided quality scores generated by proposed LQE branch.

As one key part, detection head is responsible for interested objects’ classification and localization, and followed by the one-stage detectors’ designing direction of dense prediction, false positives [5, 9, 10] or the inconsistency of the two tasks caused by feature misalignment [9, 26, 27, 31, 34, 36] or the separate design of the two tasks [1, 2, 13, 14] becomes a key problem.

As Fig.1 shows, the original detection head (a) consists of two separate branches that have a weak correlation, leading to a fact that the predicted objects’ classification score and localization quality may diverge in training and inference phase, then some poor predictions with low localization accuracy or false positives may be selected as final results during post process [16]. Then primary localization quality branch is proposed, as Fig.1(b) and (c) shows, the former utilize “centerness” score to lower poor samples’ contributions that are far away from the center of objects. The latter treats predicted IoU as localization quality, which is then combined with classification score to form a final score for the rank process of NMS. But this design brings another gap between training and inference, where some true negatives with a high localization quality score may rank in front of true positives then degrade model’s performance because of the separate and inconsistent usage of classification and localization quality branches [2].

For eliminating the above inconsistency risk while still being able to aware localization accuracy during inference, soft-label-based methods are proposed as Fig.1 (d) shows. Instead of using independent localization accuracy branch’s scores during inference, a joint score consisting of category score and samples’ IoUs are used to replace standard one-hot category label to train the classification branch, which realizes that being able to aware localization accuracy by regulating classification branch’ training. However, as this design directly takes IoU as soft labels, the classification branch will face more pressure and it could not show full strength because during inference the IoUs cease to exist. So we still believe the necessity of localization quality branch and there should exist a more reasonable instruction to generate better ranking scores.

Thus, we designed and proposed our CLQ head. As Fig.2 shows, distinct from previous soft-label-based LQE methods where QE score is constituted directly by classification score and samples’ IoUs during training, or some independent QE branch which only works during inference phase, we improved the detection head by firstly designing an IoU-guided LQE branch responsible for predicting a reliable QE score, then eliminating the gap caused by separate design and making it effective both during training and inference phase by merging the QE score with classification score. In the meantime, considering the efficiency, the LQE branch only bring a projection layer connected with regression branch, which brings a negligible computation burden to the whole model, and a sigmoid activation layer to ensure a reasonable predicted IoU value between 0 and 1.

As for the training procedure, BCE loss is used to supervise the LQE branch and positive samples are used for calculating the loss, as (1) indicates.

where $IoU_{i}^{\prime}$ is the training target defined by the overlap of current anchor box $b_{i}$ and its corresponding ground-truth box. Instead of directly using predicted IoU as QE score roughly during inference, we define the QE score using its exponential form as (2) indicates.

Secondly, for better making consistence of classification and localization quality, we adopt the joint representation to produce final ranking score (RS) to help NMS select truly good predictions, as (3) indicates.

where the $cls\_score_{i}=[C_{1},C_{1},...C_{M}],C_{i}\in[0,1]$ stands for the classification predictions, and $QE\_score_{i}$ means the corresponding i^th localization quality score. The final ranking score is taken as the input of post process for the prediction results.

The proposed detector follows one-stage detector’s design and the forward pipeline consists of a backbone to extract primary features, an FPN to build an enhanced feature pyramid, and the proposed detection head to realize dense prediction for classification and regression, producing classification results and 4-dimensional class-agnostic offsets for every feature point. The whole framework is shown in Fig.3.

Then given an image $I$, the assigner first selects positive anchors from the feature pyramid and generates corresponding targets for training, where standard ATSS [14] assigner is adopted with selecting positive samples according to statistical characteristics of objects. Then the training loss is calculated by Quality-Focal-Loss [2] and GIoU Loss [22] respectively for classification and regression branches, as (4) and (5) indicates.

where $loc_{i}$ is predicted localization, $loc_{i}^{\prime}$ is corresponding ground-truth box, $ranking\ score_{i}$ is the joint representation of classification and QE score, $cls_{i}^{\prime}$ is category label, and $QE\_target_{i}$ is the training target of QE score. For negative feature sample, the corresponding category label and $QE\_target$ are set to be zero. Finally, with the aforementioned additional LQE branch’s loss, classification and regression loss, the total loss function is calculated as (6) indicates.

As a one-stage anchor-based detector, CLQ directly predicts category scores based on features within preset anchor boxes, while compared to which the regressed bounding boxes have changed a lot, causing feature misalignment and increasing the number of false positives. As Fig.4 shows, original red anchor contains less foreground feature so classification branch tends to classify it as background. But according to the close relationship with the black ground-truth box, the regressed yellow box gets closer to the black box and should be classified as person, where the contradiction affects the performance.

To solve this problem, we further refined the CLQ head and adopted a DCN [29] component for using aligned features based on regression branch’s offset predictions. As a result, the final ranking scores consisting of more accurate classification scores and LQE scores could better indicate the localization qualities of the regressed bounding boxes, thus improves model’s performance. The structure of improved CLQ head is shown in Fig.5.

Taking the original features from FPN and densely predicted regression offsets as inputs, DCN adjusts features under the guidance of regressed boxes’ localizations to ensure the feature regions used for classification and localization quality estimation well match the corresponding regressed bounding box.

Dataset settings: We did experiments on COCO benchmark [38], where trainval35k with 115K images are used for training and minival with 5K images for validation. To demonstrate the generalization ability of our detector, the standard COCO-style Average Precision (AP) metrics are adopted.

Implementation Details: All the object detectors are implemented based on Pytorch and MMdetection [39] for fair comparison where the default hyper-parameters are adopted. Unless otherwise stated, we apply the standard 1x learning schedule (12 epochs) with a single shorter size of 800 pixels and without multi-scale training for the ablation study, based on ResNet-50 [40] as backbone.

Accurate localization quality score generation: we first explore our model with various $\alpha$ during training and the results are shown in Table.1. For generating more suitable localization quality scores, we use the hyper-parameter $\alpha$ to modulate the contributions of LQE branch. As (2) shows, where bigger $\alpha$ makes the quality score play a bigger role while vice versa. Based on Table.1, we choose $\alpha=0.3$ as optimized value, which provides a higher 0.2 AP improvement, indicating an unequal contribution of classification and LQE branch when generating the final ranking scores.

CLQ head improves detection performance: to respectively verify the effect of the designed branches in CLQ head, we further did ablation study by adding corresponding modules one by one, and results are shown in Table.2. As CLQ takes a joint representation of category score and localization quality score as final ranking score, where the standard one-hot category label is softened as continuous values, QFL [2] which is suitable for continuous category loss’s calculation is used to replace traditional Focal Loss [35]. Based on that, as Fig.2 shows, adding an LQE branch into vanilla one stage detection head in our CLQ model gives a 0.5 AP improvement with almost no sacrifice in speed because of the negligible amount of extra parameters. When further using aligned features, a total 1.7 AP improvement can be acquired at the expense of one seconds inference time, which is 23.7 fps, but this inference speed still meets most real time requirements

Good scalability of CLQ: beyond CLQ, we transfer the designed LQE branch and feature alignment module to symbolic one stage detector ATSS. Based on the results shown in Table.3, adding an LQE branch into ATSS gives a 0.4 AP improvement compared with only using QFL, and when further using aligned features, an extra 0.8 AP improvement can be reached. Totally, a 1.2 AP improvement can be provided with the help of CLQ head. More detailed, as shown in Fig. 6, the training loss curves with different epochs are drawn. We conclude the CLQ does improve detection performance, indicating its good effectiveness and scalability.

We evaluate our model on the COCO test-dev to compare with symbolic state-of-the-art convolution-based object detectors. Multi-scale training strategy and 2x learning schedule (24 epochs) are adopted. For fair comparison, the trained models

provided by MMDetectioin [39] with different backbones are evaluated as the baselines. The comparison results are shown in Table 4, our model performs well and it reaches the state-of-the-arts’ performance, and our best result with a single ResNeXt-101-64x4d model achieves considerably competitive 47.8 AP with an inference speed of 11.5 fps.

For better visualization of the speed and accuracy trade off, we graphically represent the data in Table 4, as Fig.7 shows. Because of the elegant design with extra negligible parameters, CLQ offers a significant speed advantage while maintains competitive accuracy, and produces a (AP, speed) of (44.2, 18.3), (46.2,15.1), (47.8,11.5) based on ResNet-50, ResNet-101, ResNeXt-101 respectively, showing a broader application potential by guarantying the real-time performance.

As a concluding, the designed CLQ generates a suitable localization quality score and works together with the classification branch, making the detector able to consider localization quality both during training and inference, thus improving the consistency of original independent classification and localization tasks. Moreover, the performance is further improved by using aligned features taking the ride on the coattails of deformable convolutional network. Fig.8 shows detection results of CLQ based on ResNet-101, with the designed CLQ, an accurate classification and localization ability can be reached, especially for small or densely distributed objects.

Localization quality estimation is an effective mean to help NMS select better predictions with more accurately localized bounding boxes by merging its quality score with original classification scores. While the quality score has a close relationship with IoU which relies heavily on anchors. On the other side, anchor-free counterparts usually maintain faster speed and competitive detection accuracy because of less anchor-related parameters [13] and region-based label assignment [5, 10, 30, 37, 46]. Currently, among anchor-free detectors, there is few localization quality estimation related works. Thus, improving anchor-free detectors by riding on the coattails of the localization quality estimation to further raise the envelope of accuracy-speed boundary may become another potential research. Finally, as improving the detector without sacrificing inference speed is always our goal, CLQ is proposed with an elegant LQE branch and a lightweight feature alignment branch, and we plan to convert it to anchor-free detectors to further boost the inference speed.